How to force Logstash to reparse a file?

By default logstash writes the position is last was on to a logfile which usually resides in $HOME/.sincedb. Logstash can be fooled into believing it never parsed the logfile by specifying /dev/null as sincedb_path. Here the part of the documentation Input File. Where to write the since database (keeps track of the current position of … Read more

File Uri Scheme and Relative Files

In short, a file URL takes the form of: file://localhost/absolute/path/to/file [ok] or you can omit the host (but not the slash): file:///absolute/path/to/file [ok] but not this: file://file_at_current_dir [no way] nor this: file://./file_at_current_dir [no way] I just confirmed that via Python’s urllib2.urlopen() More detail from http://en.wikipedia.org/wiki/File_URI_scheme: “file:///foo.txt” is okay, while “file://foo.txt” is not, although some interpreters … Read more

Simple way to copy a file

Warning: This answer is mainly about adding a hard link to a file, not about copying the contents. A robust and efficient copy is conceptually simple, but not simple to implement due to the need to handle a number of edge cases and system limitations that are imposed by the target operating system and it’s … Read more

Two way sync with rsync

Try Unison: http://www.cis.upenn.edu/~bcpierce/unison/ Syntax: unison dirA/ dirB/ Unison asks what to do when files are different, but you can automate the process by using the following which accepts default (nonconflicting) options: unison -auto dirA/ dirB/ unison -batch dirA/ dirB/ asks no questions at all, and writes to output how many files were ignored (because they … Read more

SHA-256 or MD5 for file integrity

Both SHA256 and MDA5 are hashing algorithms. They take your input data, in this case your file, and output a 256/128-bit number. This number is a checksum. There is no encryption taking place because an infinite number of inputs can result in the same hash value, although in reality collisions are rare. SHA256 takes somewhat … Read more

How to perform file system scanning

EDIT FOR 1.16: Enough people still hit this answer, that I thought I’d update it for Go 1.16. The function filepath.WalkDir introduced in Go 1.16 has better performance than filepath.Walk mentioned in the previous edit. Here’s a working example: package main import ( “flag” “fmt” “io/fs” “path/filepath” ) func visit(path string, di fs.DirEntry, err error) … Read more

What are .S files?

.S files are source code files written in assembly. Assembly is an extremely low-level form of programming. The files contain assembly instructions to the processor in sequential order and are typically compiled based on a selected architecture. Examples of such files are often seen in the linux kernel for specific architectures, e.g. x86, sparc, ARM, … Read more

How to get file length in Go?

(*os.File).Stat() returns a os.FileInfo value, which in turn has a Size() method. So, given a file f, the code would be akin to fi, err := f.Stat() if err != nil { // Could not obtain stat, handle error } fmt.Printf(“The file is %d bytes long”, fi.Size())