Find duplicates in a folder and convert them to symlinks ubuntu 14.04

Hi,
Duplicate files waste precious space on your computer’s hard drive.
Here a terminal approach with a script to vonvert them to symlinks:

first install the dependencies:

Code:

sudo apt-get install rdfind symlinks

Now create the blxremoveduplicates script:

Code:

#!/bin/bash
if [[ -d "$1" && ! -L "$1" ]] ; then
    rdfind -makesymlinks true $1
    symlinks -crs $1
else
echo "Enter a valid directory name!"
fi

USAGE : ./blxremoveduplicates folder_name

You can "compile" it with shc and place it in the /usr/bin folder after that.
if you know that your directory contains broken symlinks use this script instead:

Code:

#!/bin/bash
if [[ -d "$1" && ! -L "$1" ]] ; then
     find -L $1 -type l -delete
     rdfind -makesymlinks true $1
     symlinks -crs $1
else
echo "Enter a valid directory name!"
fi

Another method:

Code:

There are still good, mature applications for finding and removing duplicate files on Linux. FSlint offers a good graphical interface and should be available in most Linux distributions’ software repositories. You won’t find the shiniest interfaces, but you will find simple, functional software that doesn’t want to push junkware on you or sell you anything.

source: http://www.howtogeek.com/201167/how-to-find-and-remove-duplicate-files-on-any-operating-system/

You can install this with:

Code:

sudo apt-get install fslint

start it with: fslint-gui

Fslint is a bit easier to use but i do see that it fails very often and the method is nor so stable as my command line approach, but perhaps you`ll like fslint more, so i posted it here too.

Hope this tutorial it helped someone.

Experimental Compression method for folders on ubuntu 14.04/14.10

Hi,

Do show you here a method to compress folders, that is not recommended for very very very very and very important backups, but might be interesting to decrease file size for transferring files etc. etc. :

first you need to install lrzip tar xz-utils and sed:

Code:

sudo apt-get install lrzip tar xz-utils sed

Now create a file named blxcompress.sh or any other name you want it to have with following content:

Code:

#/bin/bash

echo Compressing folder $1

tar cvf $1.tar $1

lrzip -n -o $1.tlrz $1.tar

xz -z $1.tlrz

rm $1.tar

Now you can compress a folder by typing ./blxcompress.sh folder_name

If you want to decompress create the following ./blxdecompress.sh script:

Code:

#/bin/bash

fname=`echo "$1" | sed 's/........$//'`

echo You compressed the folder $fname

cp $fname.tlrz.xz $fname-copy.tlrz.xz

xz -d $fname-copy.tlrz.xz

lrzip -d -o $fname-copy.tar $fname-copy.tlrz

tar -xvf $fname-copy.tar

rm $fname-copy.tar
rm $fname-copy.tlrz

echo folder $fname extracted sucessfully

Usage: ./blxdecompress.sh folder_name.tlrz.xz

This compression method may be better than normal xz compression. The reason is that it does search for repeating patterns in a distance of several hundreds of MB. Normal gzip and xz can`t do this.

lrz is a good compression format if you ahve files that contents repeat very often. This method does devide the process up in three steps and that is more stable than letting lrzip do all. lrzip gets really slow on my on if i try to use itas xz method. If i do use xz after lrzip it does work very well.

Tought it is ideal for compression of 1-4 GB files and if you have time for compression and decompression since its really slow but has sometimes nearly the same ratio as normal zpaq which is much slower.

You might "compile these scripts with shc and put them in /usr/bin if you like to do that!

regards bluedxca93

Image manipulating with lua5.1 -imlib2 on ubuntu 14.04

Hi,

Gmic is a bit complicated, gluas has to many bugs and imagemagick is in C, but you can`t write C programs and need to write your own simple image filter?
Then the following instruction might be intertesting if you have ubuntu installed:

Lua is implemented in pure ANSI C and compiles unmodified in all platforms that have an ANSI C compiler. Lua also compiles cleanly as C++. source:
http://www.lua.org

How to install lua5.1 and the imlib2 plugin :

Code:

sudo apt-get remove  lua5.2
sudo apt-get install  lua5.1  luarocks libimlib2-dev
sudo luarocks install lua-imlib2

Reference for easier writing your own image filter you `ll find here:

http://asbradbury.org/projects/lua-imlib2/doc/

However i didn`t understand somje things of it at the beginning so im now posting a sample script:

Code:

-- simple selective blur
-- by bluedxca93

require("imlib2")


local im =  imlib2.image.load("input.png")
local xlength = im:get_width()
local xheight = im:get_height()

for x=0, xlength-1 do
  for y=0, xheight-1 do
     level = 15
     c0 = im:get_pixel(x, y)
     c1 = im:get_pixel(x-1, y-1)
     c2 = im:get_pixel(x, y-1)
     c3 = im:get_pixel(x+1, y-1)
     c4 = im:get_pixel(x-1, y)
     c5 = im:get_pixel(x+1, y)
     c6 = im:get_pixel(x-1, y+1)
     c7 = im:get_pixel(x-1, y+1)
     c8 = im:get_pixel(x-1, y+1)

      c0r = (c0.red)  
      c1r = (c1.red)
      c2r = (c2.red)
      c3r = (c3.red)
      c4r = (c4.red)
      c5r = (c5.red)
      c6r = (c6.red)
      c7r = (c7.red)
      c8r = (c8.red)
         
      c0g = (c0.green)
      c1g = (c1.green)
      c2g = (c2.green)
      c3g = (c3.green)
      c4g = (c4.green)
      c5g = (c5.green)
      c6g = (c6.green)
      c7g = (c7.green)
      c8g = (c8.green)
 
      c0b = (c0.blue)
      c1b = (c1.blue)
      c2b = (c2.blue)
      c3b = (c3.blue)
      c4b = (c4.blue)
      c5b = (c5.blue)
      c6b = (c6.blue)
      c7b = (c7.blue)
      c8b = (c8.blue)
       
          if c0r > c1r then cv1r = c0r - c1r else cv1r = c1r - c0r end
          if c0r > c2r then cv2r = c0r - c2r else cv2r = c2r - c0r end
          if c0r > c3r then cv3r = c0r - c3r else cv3r = c3r - c0r end
          if c0r > c4r then cv4r = c0r - c4r else cv4r = c4r - c0r end
          if c0r > c5r then cv5r = c0r - c5r else cv5r = c5r - c0r end
          if c0r > c6r then cv6r = c0r - c6r else cv6r = c6r - c0r end
          if c0r > c7r then cv7r = c0r - c7r else cv7r = c7r - c0r end
          if c0r > c8r then cv8r = c0r - c8r else cv8r = c8r - c0r end

          if c0g > c1g then cv1g = c0g - c1g else cv1g = c1g - c0g end
          if c0g > c2g then cv2g = c0g - c2g else cv2g = c2g - c0g end
          if c0g > c3g then cv3g = c0g - c3g else cv3g = c3g - c0g end
          if c0g > c4g then cv4g = c0g - c4g else cv4g = c4g - c0g end
          if c0g > c5g then cv5g = c0g - c5g else cv5g = c5g - c0g end
          if c0g > c6g then cv6g = c0g - c6g else cv6g = c6g - c0g end
          if c0g > c7g then cv7g = c0g - c7g else cv7g = c7g - c0g end
          if c0g > c8g then cv8g = c0g - c8g else cv8g = c8g - c0g end

          if c0b > c1b then cv1b = c0b - c1b else cv1b = c1b - c0b end
          if c0b > c2b then cv2b = c0b - c2b else cv2b = c2b - c0b end
          if c0b > c3b then cv3b = c0b - c3b else cv3b = c3b - c0b end
          if c0b > c4b then cv4b = c0b - c4b else cv4b = c4b - c0b end
          if c0b > c5b then cv5b = c0b - c5b else cv5b = c5b - c0b end
          if c0b > c6b then cv6b = c0b - c6b else cv6b = c6b - c0b end
          if c0b > c7b then cv7b = c0b - c7b else cv7b = c7b - c0b end
          if c0b > c8b then cv8b = c0b - c8b else cv8b = c8b - c0b end

            if level > cv1r and level > cv2r and level > cv3r and level > cv4r and level > cv5r and level > cv6r and level > cv7r and level > cv8r  then   dr = ((c1r + c2r + c3r + c4r + c5r + c6r + c7r + c8r )/8 ) else dr = c0r end

            if level > cv1g and level > cv2g and level > cv3g and level > cv4g and level > cv5g and level > cv6g and level > cv7g and level > cv8g  then   dg = ((c1g + c2g + c3g + c4g + c5g + c6g + c7g + c8g )/8 ) else dg = c0g end

            if level > cv1b and level > cv2b and level > cv3b and level > cv4b and level > cv5b and level > cv6b and level > cv7b and level > cv8b  then   db = ((c1b + c2b + c3b + c4b + c5b + c6b + c7b + c8b )/8 ) else db = c0b end

              cf = imlib2.color.new( dr, dg, db)
     
  im:draw_pixel(x, y, cf)
  end
end


im:save("output.png")

file name: blxblur.lua
Usage lua blxblur.lua reads input.png and prints output.png

Writing filters is easy, but the required text, to do it is very long, but with some simple find and replace command and a logical way to write them, you don`t type that much. believe me. I thought the code is long, but it schould help people to learn how to write their first image processing filters and thats why i posted it in full length!.
It is human readbale code you do use for source coede and the lua interpreter will help you to find errors more quickly than any other meothod ive tried so far. Its ten times faster than using bash for doing it but also five times slower than a imagemagick function, however quickly enough to convert images in a reasonable amount of time.

regards bluedxca93