Hacker Newsletter - a weekly newsletter of the best articles on startups, programming, and more. All links are curated by hand from Hacker News.
Announcing my first e-book "Awk One-Liners Explained"
Buy it now for just $5.95
Hello everyone! I have awesome news - I have written my first e-book ever! It's called "Awk One-Liners Explained" and it's based on the "Famous Awk One-Liners Explained" article series that I wrote here on my blog a few years ago and that has been read over 800,000 times!
I went through all the one-liners in the article series, improved the explanations, fixed the mistakes, added an introduction chapter to Awk one-liners, and two new chapters on the most commonly used Awk special variables and on idiomatic Awk.
Table of Contents
The e-book is 58 pages long and contains exactly 70 well-explained Awk one-liners. It's divided into the following chapters:
- Preface.
- 1. Introduction to Awk One-Liners.
- 2. Line Spacing.
- 3. Numbering and Calculations.
- 4. Text Conversion and Substitution.
- 5. Selective Printing and Deleting of Certain Lines.
- 6. String and Array Creation.
- Appendix A: Awk Special Variables.
- Appendix B: Idiomatic Awk.
- Index.
What is awk?
Awk is this awesome, little program that's present on nearly ever Unix machine. It's designed to carry out various text processing tasks easily, such as numbering lines, replacing certain words, deleting and printing certain lines.
Let's take a look at several examples.
Example 1: Print the second column from a file
awk '{ print $2 }'
That's all there is to it. Awk automatically splits each line into columns and puts each column in variables $1, $2, $3, etc. This one-liner prints just the 2nd column, which is in variable $2.
You can also specify the symbol or word that you wish to split on with the -F command line switch. This switch is explained in more details in the e-book and in the last example below.
Example 2: Number lines in a file
awk '{ print NR ": " $0 }' file
The whole line itself goes into variable $0. This one-liner prints it but prepends the NR special variable and a colon ": " before it. The special variable NR always contains the current line number.
There are many other special variables and they're all explained in the e-book and summarized in the appendix.
Example 3: Count the number of words in a file
awk '{ total = total + NF } END { print total }'
Here another special variable is used. It's the NF that stands for number of fields, or number of columns, or number of words in the current line. This one-liner then just sums the total number of words up and prints them before quitting in the END block.
Example 4: Print only lines shorter than 64 characters
awk 'length < 64'
This one-liner uses the length function to determine the length of the current line. If the current line is less than 64 characters in length, then length < 64 evaluates to true that instructs awk to print the line.
Finally, let's take a look at an example that compares an Awk program with an equivalent C program. Suppose you want to print the list of all users on your system. With Awk it's as simple as this one-liner:
awk -F: '{ print $1 }' /etc/passwd
This one-liner says, "Take each line from /etc/passwd, split it on the colon and print the first field of each line." Very straightforward and easy to write once you know Awk!
Suppose you didn't know Awk. Then you'd have to write it in some other language, such as C. Compare the example above with the example in C language:
#include <stdio.h> #include <stdlib.h> #include <string.h> #define MAX_LINE_LEN 1024 int main () { char line [MAX_LINE_LEN]; FILE *in = fopen ("/etc/passwd", "r"); if (!in) exit (EXIT_FAILURE); while (fgets(line, MAX_LINE_LEN, in) != NULL) { char *sep = strchr(line , ':'); if (!sep) exit (EXIT_FAILURE); *sep = '\0'; printf("%s\n", line); } fclose(in); return EXIT_SUCCESS ; }
This is much longer and you have to compile the program first, only then you can run it. If you make any mistakes, you have to recompile it again. That's why one-liners are called one-liners! They are short, easy to write and they do one thing really well. I am pretty sure you're starting to see how mastering Awk and one-liners can make you much more efficient when working in the shell and with text files in general.
Once you read the e-book and work through the examples, you'll be able to solve the most common text processing tasks, such as joining lines in a file, numbering lines, replacing certain words and printing certain lines.
Book preview
I prepared a book preview that contains the first 11 pages of the book. It includes the table of contents, preface, introduction and the first page of the second chapter.
Awk One-Liners Explained. Book Preview.
Buy it now!
The price of the e-book is just $5.95 and you can buy it through PayPal.
After you have made the payment, my automated e-book processing system will send the PDF e-book to you in a few minutes!
Testimonials
Iain Dooley, CEO and founder of Working Software LTD:
It never ceases to amaze me that, even though I spend 50% - 70% of my day using a *nix command line and have done so for the past 6 years, there are still countless thousands of useful tools and tips to learn, each with their corresponding little productivity boosts. The trouble, of course, is finding the time to organise and prioritise them, deciding which are the most important and useful to learn. "Awk One Liners Explained" is a fantastic resource of curated examples that helped me rapidly pick up a few cool tricks that have already provided many times the value I initially paid for the book. Any professional who spends time working with *nix systems can benefit from this book.
Tweet about my book!
I am really excited about my book and I would appreciate your help spreading the word via Twitter! Here is a quick link for tweeting:
My other e-books!
I am so passionate about programming and writing about programming that I have now written my second e-book called "Sed One-Liners Explained". It's written in the same style as this e-book and it explains sed, the Superman of Unix stream editing. Sed One-Liners Explained contains 100 well-explained one-liners and it's 98 pages long. Take a look!
And I am not stopping here - I am going to release several other books. My next e-book is called "Perl One-Liners Explained" and it's based on my "Famous Perl One-Liners Explained" article series.
Enjoy!
Enjoy the book and let me know what you think about it!
Sincerely,
Peteris Krumins (@pkrumins on Twitter)
Related Posts
- Announcing my third e-book "Perl One-Liners Explained"
- Announcing my second e-book "Sed One-Liners Explained"
- Awk, Nawk and GNU Awk Cheat Sheet
- Update on Famous Awk One-Liners Explained: String and Array Creation
- Famous Awk One-Liners Explained, Part III: Selective Printing and Deleting of Certain Lines
- Famous Sed One-Liners Explained, Part III: Selective Deletion of Certain Lines and Special Applications
- Bash Emacs Editing Mode Cheat Sheet
- Working Productively in Bash's Vi Command Line Editing Mode (with Cheat Sheet)
- Famous Awk One-Liners Explained, Part II: Text Conversion and Substitution
- Famous Awk One-Liners Explained, Part I: File Spacing, Numbering and Calculations
Comments
I'd rather buy it if it wasn't only in PDF format... (.mobi || .prc prefered, or I can help you to make a conversion if needed)
Is there a chance?
Thanks for the question. At the moment I only have the PDF but , yes, I'll have those formats available later. I'll announce it here on my blog when I have them ready!
Thanks for Your answer Peteris, then my next question is: if I order the PDF, will I get the other (Kindle (prc||mobi)) version for free?
If you buy the book right now, I'll send you the other version for free. Otherwise, it will be available for purchase on Kindle store!
But note that it will take me some time before I convert the book into those formats!
Fair enough! Order placed. May I recommend to try out Calibre [for conversion] which does a great job especially from HTML (or markdown) input? Or if you have a LaTeX source, there's pandoc which happily translates to several output format AFAIK.
Payment received, and my book system just sent you the book as attachment. If you don't see it in your inbox, please check your spam folder. (If still nothing, mail me!!!).
I have LaTeX source, I am gonna check out pandoc, or just prepare the document dimensions for Kindle (from their guidelines).
Some error reporting (but still, thanks for the great tutorial):
- @2.2: awk 'BEGIN { ORS="\n\n" }; 1' the ; is not necessary AFAIK.
- @3.4: same applies in awk 'NF { $0=++ a " : " $0 }; { print }'
- @3.7: ; before END
- @3.9-3.11,3.14: ; before END
- @4,17: "This is the ο¬rst time we see the -F argument passed to Awk." Actually we had seen in @1.1.
Thanks for finding and reporting the errors!
There is nothing wrong with having ;, but yes, you're right, and they can be left out! One char shorter. :)
@4.17 - good catch!
Wow! This is just what I need. I'm definitely going to buy a copy. But I've got to ask, will you have a version for Nook or the Sony Reader? I'm buying either of the two and of course I want to seo have a copy of your e-book uploaded into it. Thanks! Oh, congratulations!
I just bought the book. It looks very interesting. I would really like a .mobi version too.
Thanks for buying the book! I'll send you a .mobi for free when it's out!
Is this ebook available on the Amazon kindle store?
It's not yet available on the Amazon kindle store. I only have PDF for sale right now. I have planned on publishing a Kindle version later. I'll announce it on my blog when it's ready!
Kindle will display .pdf well enough, though one usually rotates the screen for good justice.
Thanks for your efforts, Peteris!
You're welcome! Thanks for buying a copy. :)
Talking about Kindle, I'd like it to be really fit for Kindle, so I'll take time to prepare a version just for Kindle.
Any progress on the Kindle version?
Yes, I wrote a parser that converts my LaTeX documents to Kindle. Now I need to format them nicely.
Wow! This is just what I need. I'm definitely going to buy a copy. But I've got to ask, will you have a version for Nook or the Sony Reader? I'm buying either of the two and of course I want to have a copy of your e-book uploaded into it. Thanks! Oh, congratulations!
Thank you! Yes, I'll have a version for Nook and Sony Reader but that will take some time to make. I don't have an estimate but I'll announce it on my blog when it's ready!
Nice! I like pdf and html, but will read epub with a FireFox addon (www.epubread.com).
Take a look at asciidoc. Documents, (e)books, etc., are written in text and than translated to html, pdf, epub, docbook, etc. I use it for documentation and notes, including math notes (w/the help of asciimathml js). Write it once in text without a lot of messy markup, but the final output can be any format. Good example at the link below --- although it supports most ebook readers, it looks like it doesn't support kindle.
www.methods.co.nz/asciidoc/publishing-ebooks-with-asciidoc.html
I wrote it in LaTeX. Looks like it will be trickier to convert it to epub, but still shouldn't be too difficult. LaTeX is LateX afterall.
Looks like it'll be great. I went ahead and paid for it on PayPal but have not yet received a link. Do you know how quickly that happens (it was a bit over an hour ago).
Hi Peter,
I just looked at my book processing system and I see that the book was generated for you:
[2011-55-20 16:55:23] Processing new payment for awk_book from pkeane@[masked]. Preparing Awk Book... Spawning Latex... Spawning makeindex... Spawning Latex 2nd time... Spawning Latex 3nd time... Sending the Awk book to Peter Keane (pkeane@[masked]).
And that it was sent for you successfully (your mail server accepted it):
Jun 20 11:55:30 catonmat sm-mta[789]: p5KGtThY000786: to=<pkeane@[masked]>, delay=00:00:00, xdelay=00:00:00, mailer=esmtp, pri=641910, relay=inbound.mail.utexas.edu. [146.6.25.62], dsn=2.0.0, stat=Sent (ok: Message 314653467 accepted)
It looks like your mail server didn't like that email for some reason and didn't deliver it further to you. Or if it did, can you check the spam folder? One other person who bought the book told me he found the book in his spam folder!
In any case, I just generated the book for you sent you an email with the book attached personally. Please confirm that you have received it.
And huge thanks for purchasing my book!
Update:
I just received an email from Peter, and indeed, the email (in fact both of them) were in the spam folder. Here is his response:
Yes -- there it is in my spam folder! (2 messages, in fact). Sorry for the trouble & thanks for the book.
So for everyone who purchases the book - if you don't receive it in 10 mins, please check the spam folder, it must be there!
Maybe you can use some help with DKIM and more email white-listing techniques?
Congratulations pal on the new release, best of luck.
Thanks and this sounds exactly what I need. Let's talk!
Thank you! Could you please elaborate a bit on your hack "Prepared exclusively for foo (email address)"
Sure. The ebook is written in LaTeX and I use the background package with the following configuration options to add the message to the book:
\usepackage{background} \SetBgContents{Prepared exclusively for Peteris Krumins (peter@catonmat.net)} \SetBgAngle{0} \SetBgScale{1} \SetBgOpacity{1} \SetBgColor{preparedfor} \SetBgPosition{current page.south} \SetBgVshift{0.5cm}
The preparedfor color is defined by the help of xcolor package:
\usepackage{xcolor} \definecolor{preparedfor}{HTML}{787878}
I could not get your web page to load with Internet Explorer. Also what's up with the "congradulations you are the 100,000th visitor" at the bottom of the page? Must be some ad. It is annoying, because I hear the "congradulations" when I first load up your page in Chrome.
Besides all this, good luck with your book. Are you going to publish a hard copy version?
Sorry about that nasty ad. My ad company promised that such ads won't appear on my site. Looks like they broke their promise.
I have plans to publish the hard copy