A Collection of UNIX Commands I Find Useful

1. Deleting recursive files & directory structure

find . -name "FILE-TO-FIND"-exec rm -rf {} \;

b. Deleting recursive files but leaving directory structure

find . -type f -name "FILE-TO-FIND" -exec rm -f {} \;

c. finding empty directories and deleting them

find . -type d -empty -exec rmdir {} \;

d. deleting files of a certain size

 find . -size <size>c -exec rm {} \; --where c=kb M=MB, etc

Note: if you want less than a specific size for <size> put (eg) -20M or more +20M

e. Removes all files older than and including <file>

find . ! -newer <file> -exec rm -rf {} \;


2. to display filename and line count minus 1 for csv files (well, any text file)

for i in `ls -1`; do echo -n "$i,"; j=`cat $i|wc -l`;j=`expr $j - 1`;echo $j; done

or

wc -l * | awk '{ if ( $2 != "total" ) printf"%s,%s\n",$2,$1-1}'


3. to remove whitespace and replace with underscores from file names recursively

NOTE: this method is currently imperfect if subdirectories also contain white space

for i in `find . -name '* *' | tr ' ' '~'`; do mv "$(echo $i| tr '~' ' '|sed 's/ /\\ /g')" "$(echo $i | tr '~' '_')"; done


4. how to flatten a directory structure

find . -type f -name "*mp3" -print0 | xargs -0 -J% mv % <destination_directory>

This example scans a directory tree for all mp3 files and moves them to a single directory


5. display full paths to files for recursive directories

find `pwd` -name "*.*"


6. copying a disk to a disk image using dd

dd bs=512 if=/dev/rXX# of=/some_dir/foo.dmg conv=noerror,sync

NOTE: conv=noerror,sync skips bad blocks and fills the target with nulls, thereby making it an excellent way to recover a broken disk, so long as it mounts.


The basic syntax for dd is:

dd if=INPUT-FILE-NAME of=OUTPUT-FILE-NAME

b. making an ISO image using dd

To make an ISO from your CD/DVD, place the media in your drive but do not mount it. If it automounts, unmount it.

 dd if=/dev/dvd of=dvd.iso    # for dvd

dd if=/dev/cdrom of=cd.iso bs=2048  # for cdrom

dd if=/dev/scd0 of=cd.iso    # if cdrom is scsi


c. burning disk image to disk / usb using dd

This is useful for making bootable USB Sticks of OS iso or dmg images

dd if=<path_to_image>/<imagefile.dmg of=/dev/rdisk8 bs=1m

NOTE: Remember to write it to the raw disk.


7. Symlinks 

a. Find broken symlinks

find . -type l -print -exec file {} +|grep broken

b. show target and source of symlinks in a dir

find . -type l -exec readlink {} \; -exec echo {} \;|awk '{printf"%s - ", $0 ; getline ; print $0 }'


8. Remove spaces from a file name recursively and replace with underscores

find . -name '* *' | while read file;

do

target=`echo "$file" | sed 's/ /_/g'`;

echo "Renaming '$file' to '$target'";

mv "$file" "$target";

done;


flattening a dir - note -print0 is mac specific to handle spaces in file names

----

find . -type f -name "*vsd" -print0|xargs -0 -J% cp % /Volumes/Charles05/charles/cw/customers/vertex/One/design/visio_vdx 


gzip and find

-------------

find . -type f -name "archlog*.dbf" -exec gzip {} \; &        # find everything

called archlog*.dbf and then compress it


or


find . -type f -name "archlog*.dbf" -mtime +2  -exec gzip {} \; &    # find

everything older than two days and compress it


or


gzip archlog*.dbf &      # Compress all logs with this name .


---------------------


cd <dir>

for f in `ls arch*.dbf`

do

if [ -z `fuser $f` ] ; then

   gzip $f

else

   echo "$f is in use"

fi


or this:


cd <archdir>

find . -type f ! -newer `ls -1tr | tail -1` -xargs gzip {} \;



gzip a directory to a single file

---------------


tar cvf - <directory to be zipped>|gzip > <filename of single file to compress to>.gz



grepping for multiple words use egrep

---------------------------------------


egrep "(<word1>|<word2>)" filename


e.g. egrep "(CREATE|DROP)" IS001A.sql



full fsck

---------

fsck -F vxfs -o full -y <File system>


e.g. fsck -F vxfs -o full -y /dev/vx/dsk/bcv1OtherDbs-livedg/OtherDbs-orabin-9i



Keeping the sudo token active

------------------------------


while true

do

sudo ssh crockofshi7

sleep 180

done


while true; do sudo ssh thingy; sleep 240; done


Finding how many processors

--------


psrinfo


Finding out where the large files are

--------


du -sk *


pkill

-----


killing processes by name


pkill -9 <string>


adding users

------------


useradd -g 10 -d /export/home/charlesisapoof -m -s /bin/ksh -c "Charles Webster" websterc



back-to-back-tar

-----------------


How to push a whole file system from one machine to another through ssh


tar cf - * | ssh -c blowfish file_share@tiger "cd /Users/file_share/jad; tar xf -"


full path listings

-------------


 for i in `ls -1`;do echo `pwd`\/$i; done


or in awk (courtesy of Dave Leather)


ls|awk '{print "'`pwd`'/" $1}


if you want to exclude certain files based on file name use this (example here is "small")


ls *.jpg|awk '{if ( $0 !~ /small/) print "'`pwd`'/" $1}


(to do this on inclusion rather than exclusion drop the !)


-----------


Sun inventory

--------


/usr/platform/SUNW,Sun-Fire-V245/sbin/prtdiag


Making an encrypted disk image (sparsefile)

----------------------


hdiutil create -size 62gb -encryption -type SPARSEBUNDLE -fs "Case-sensitive Journaled HFS+" -volname "Charles08"  /Volumes/CSW_USB_08/Charles08.sparsebundle -spotlight




converting sparseimage to sparsebundle

-----------------------------

hdiutil convert -format UDSB -o <target.sparsebundle> <source.sparseimage>



------


Solaris - Displaying kernel parameters not listed is /etc/system

---------------------------------------------------


sysdef



script to read a log file showing the current and a backup of a file and report the differences

----------------------------------


ls -t /u01/app/oracle/product/istore/cwciteappl/admin/CWCITE_ordsvywat-sun-09/log/*/adconfig.log | head -1 | \

awk '{ printf" grep backup %s\n",$0 }' |sh  |\

awk '{print "cmp " $3 " " $5 }' | sh



LINUX with HP RAID Controller - Checking status of HP Raid Controller

-----------------------------

/usr/sbin/hpacucli controller slot=0 pd all show



mounting a SMB / NFS drive

---------------


mount -t smbfs <smb_drive_to_mount_with_auth> <local_mont_location>


mount -t smbfs //cwebster@bdc.localnet/CSW01 /<path_to_mount_point/<mount_point>


on osx

--


mount_smbfs //cwebster@bdc.localnet/CSW01 /<path_to_mount_point/<mount_point>


#to mount nfs stipulate nfs rather than smbfs





Encrypt a file using OpenSSL 

-----------------------------


This is a pretty simple way to encrypt a single file so that it is nearly impossible for others to read, and it will require a password to access again. You’ll need to use the OpenSSL technology via the command line for this to work.


Encrypt the file:

$ openssl enc -e -a -salt -bf -in <plain_filename.txt> -out <encrypt_filename.txt>

enter bf-cbc encryption password:

Verifying password - enter bf-cbc encryption password:


decrypt file:

$ openssl enc -d -a -bf -in <encrypt_filename.txt> -out <plain_filename.txt>

enter bf-cbc decryption password:


--find all the instances of a pattern  in excel files and pipe them to a file

find . -name '*.xls'


--find all the instances of a pattern  in excel files and pipe them to a file with associated file name

find . -name '*.xls' -exec grep -b -n -i vtx {} /dev/null ';' | strings | egrep 'vx|VX|.xls:' | awk '{if ( $0 ~ /.xls:/ ) { file=sprintf("%s",substr($0,1,index($0,":"))) ;getline} ; print file,$0 }'


Port Scanning

---------------


scan a range of servers for a specific port

----------------------

for i in $(seq 160 170); do nc -v -n -z -w 1 10.196.52.$i 389; done


Scan a server for a specific port

-----------------------


nc -z gbcwswiep005.ad.plc.cwintra.com 100-700


Finding other machines on your network (same broadcast domain)

---------------

--find the broadcast address -- ifconfig -a | grep broadcast

--ping it --ping -i 5 -c 2 <broadcast_address>



--alternatively arp -a will give you all the machines your computer has perviously interacted with.


© Charles Webster 2016