Merging Terabytes of Files - Part 1

Problem: Given terabytes of data, in millions of files, on multiple drives merging this dataset is a daunting task. 

Solution

There are several things that need to happen.

1. Duplicate files need to be identified, even if they have different names and dates.
2. Duplicate directories need to be found,  even if the directories, subdirectories, and files have different names.
3. Similar directories should be merged and naming differences reconciled.

This problem can be broken down in to three steps. The first is to build a catalog of the directories and files. The second is to process the catalog to find duplicates and similarities. The third and final step is to build the new, merged file system.

Step #1 - Cataloging the file system

To catalog the files the following is done.

1. Each directory is crawled to collect the following
• filename
• file extension
• full file path
• md5 hash (to uniquely identify the file)
• creation date
• modification date
• file size
2. A text output file is generated with all of this data

The Cataloging Script

For python 3.8+, the following script will build a catalog with path, names, size, dates, and has for each file. Once created, it easy to load this file in Excel and review the catalog.

import glob
import os
import pathlib
import datetime
import hashlib

# starting points for the catalog
root_dir = 'D:/' 
# file to store the catalog
outfilename = "Directory-Report.txt"

with open(outfilename, 'w') as outfile:

    for fullfilename in glob.iglob(root_dir + '**/**', recursive=True):
         print(fullfilename)
         
         # try:
         if True:
            path, filenameext = os.path.split(fullfilename)
            filename, file_extension = os.path.splitext(filenameext)
             
            # get md5 hash
            try:
                with open(fullfilename, "rb") as f:
                    file_hash = hashlib.md5()
                    while chunk := f.read(8192):
                         file_hash.update(chunk)
                         file_hash_str = file_hash.hexdigest()
            except:
                file_hash_str = ""


            # get create date
            fname = pathlib.Path(fullfilename)
            createdate = datetime.datetime.fromtimestamp(fname.stat().st_ctime)
            # get modification date
            moddate = datetime.datetime.fromtimestamp(fname.stat().st_mtime)
            # get file size
            size = os.path.getsize(fullfilename)
                         
             outfile.write('"%s","%s","%s","%s","%s",%s,%s,%s,%s\n' % 
                   (fullfilename,path,filenameext,filename,file_extension,createdate,moddate,size,file_hash_str))
                   
                   

Random Notes: Thoughts on metrics for engineering tools

• If training is required for successful usage, what is the average half-life of the training. In other words, if a group of users is trained, how long until 50% of the users will forget some key aspect of tool usage which drives them to abandon the tool?
• What is the average time for user to need to go to the help files to complete a task if they do not use the tool constantly?
• Can a user successfully use the tool without training?
• Are the documentation and examples sufficient for self learning?
• How many actions are required to complete a ‘quickstart’ example?
• How many decisions are required to complete a ‘quickstart’ example?
• How many choices are the in each decision in a typical workflow?
• How difficult is it to integrate the tool into automated work flows?
• How difficult is it to customize the tool?
  • Can a power user customize the tool?
• How long does it take to introduce a new feature in the tool?
• How many sentences does it take to describe why a user should adopt the tool?
• In the absence of process enforcement, would the users naturally adopt this solution?
• What is the time saving for the individual, team, and organization from the adoption of the tool?
• If the tool reduces error rates, is there feedback to the users to help them understand the improvement?
• Can the input and output to the tool be reused so that the effort can be reapplied?
• What is the ‘activation potential’ to get a new user to adopt the tool?
• Do new users request access to the tool?
• In a corporate setting, how difficult are the permissions to manage?
• If a new user if not setup, will the team be able to duplicate the permissions with without calling the developers?

This work is licensed under a Creative Commons Attribution By license.

Organizational Incentives and Responsibilities: An Observation

In a large organization, properly aligning incentives and responsibilities is always a challenge. However, sometimes it all comes together like a grand plan. Saw a situation today that demonstrated this. A year ago, a particular facility was very undesirable:  old paint, old carpets, intermittent wireless, worn furniture, etc. In a large organization, fixing mundane issues like this can be a royal challenge: forms, approvals, policy, and more. By chance (at least to a distant observer), the machinery of the organization moved people responsible for facilities into the area. One year later… new paint, new carpets, new furniture. The other groups in the area had their productivity and satisfaction improved. Responsibilities met incentives and action was taken.

This work is licensed under a Creative Commons Attribution By license.