Posts on Trickster Dev

Advanced network traffic interception techniques with mitmproxy

Wed, 03 Dec 2025 00:00:00 +0000

The most common way to use mitmproxy for API traffic interception is to use it in a default forward proxy mode. One would run mitmproxy server on a laptop or desktop computer, then configure proxy settings and install X.509 certificate on a client device (e.g. smart phone). But there is more to mitmproxy than that. It can also be used as reverse proxy, transparent proxy (on Linux and macOS), Wireguard VPN server that also intercepts network traffic, SOCKS proxy server, customisable DNS server and even as a virtual network interface on Linux. In this article we will go through some of the lesser known mitmproxy features for uncovering what goes on between servers and clients.

Reverse proxy mode

In some cases you have a server that you want to spy on, or for some reason setting up proxy configuration on the client is not viable for your use case. To cover situations like this, mitmproxy allows you to run in reverse proxy mode. Client would access mitmproxy as if it was a real server, but mitmproxy would act as a spy in the middle between client and the server being “protected”. This is similar to a common use case of nginx server, except that it is being done for the purpose of network interception. Running mitmproxy in reverse proxy mode is rather simple on a Linux server:

$ mitmproxy --mode reverse:https://trickster.dev@443 --set block_global=false

(I ran into X.509 certificate issues when trying to do this locally on macOS. If you run mitmproxy server locally --set block_global=false is not needed.)

However, there are things that we still must do on client side. First, we must override DNS name of the proxied target so that it resolves to proxy IP address. On Linux and Unix systems the simplest way to do so is to add a single line to /etc/hosts file, such as:

165.232.168.159 trickster.dev

But now there’s a matter of setting up X.509 certificate generated by mitmproxy so that it is trusted by client application(s). For example, Chrome won’t let us access the proxied site if we don’t address this matter.

Screenshot 1

To download the certificate from mitmproxy, click the GUI element saying “Not Secure” in red, then choose “Certificate Details” out of the drop down menu.

Screenshot 2

A modal view will present information about certificate. In Details tab we can verify that it is indeed issued for the target domain and press “Export…” button to download it.

Screenshot 3

Now we need to go to chrome://certificate-manager, press the “Installed by you…” table row, then press the “Import” button next to “Trusted certificates”. The GUI will present a window to pick a file from local file system and we need to pick a certificate file we downloaded earlier.

Screenshot 4 Screenshot 5

Once we do all of this, we can use the original target address to access the site and have HTTP flows intercepted for further analysis.

Transparent proxy mode

Transparent mode involves rerouting traffic at network (IP) layer so that applications without any way to configure network proxying can be covered by traffic interception. Instructions on how to do this are provided on mitmproxy documentation page.

mitmproxy as Wireguard VPN server

The easiest way to use Wireguard mode is to choose mitmproxy web UI option:

$ mitmweb --mode wireguard

In Capture tab we will have QR code we can scan with the official WireGuard app, which will guide us through a flow of setting it up on a mobile device. However, before you do that it is best to have X.509 certificate installed by going through the default mode setup procedure or some other way, so that TLS flows can be properly intercepted.

Screenshot 6 Screenshot 7

Ff you want to use TUI instead, the WireGuard configuration will be available at ~/.mitmproxy/wireguard.conf.

This mode has an added benefit of capturing not only HTTP(S) flows, but also covering DNS and some other protocols.

SOCKS proxy mode

$ mitmproxy --mode socks5

This will launch mitmproxy as SOCKS5 server, bound to a standard port - 1080. We can try it with curl:

$ curl --socks5 localhost --head https://nike.com -k -v
* Host localhost:1080 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]:1080...
* Host nike.com:443 was resolved.
* IPv6: (none)
* IPv4: 13.226.69.50, 13.226.69.57, 13.226.69.98, 13.226.69.103
* SOCKS5 connect to 13.226.69.50:443 (locally resolved)
* SOCKS5 request granted.
* Connected to localhost () port 1080
<...>

mitmproxy as DNS server

We can also use mitmproxy as a simple DNS server that implements a subset of the protocol (only A and AAAA queries) by running it in DNS mode:

$ mitmproxy --mode dns@5333

We can test it by using dnsx from ProjectDiscovery:

$ echo "google.com" | dnsx -resp -a -resolver "127.0.0.1:5333"    

      _             __  __
   __| | _ __   ___ \ \/ /
  / _' || '_ \ / __| \  / 
 | (_| || | | |\__ \ /  \ 
  \__,_||_| |_||___//_/\_\

		projectdiscovery.io

[INF] Current dnsx version 1.2.2 (latest)
google.com [A] [172.217.25.78]

Screenshot 8

Passing traffic to upstream proxy

In some cases, such as dealing with georestricted APIs, one may want to introduce a second hop to some other proxy and pass the requests there. For example:

$ mitmproxy --mode upstream:http://brd.superproxy.io:33335 --upstream-auth [REDACTED]:[REDACTED]

Value for --upstream-auth (username and password) is formatted in the same way as --proxy-auth of curl.

mitmproxy virtual network interface

This one will require Linux system with root access. It entails creating a virtual network interface that intercepts all traffic it passes through. For example:

$ mitmproxy --mode tun

Some programs, such as curl will let you specify a network interface it will use:

$ curl --interface tun0 --head https://nike.com -v -k

For others you will need to modify a routing table to make interception happen.

Screenshot 9

Scraping Next.js web sites in 2025

Fri, 31 Oct 2025 00:00:00 +0000

When looking into some targets for web scraping, you may come across pages that contain a lot of data represented in JSON-like (but not quite JSON) format passed to self.__next_f.push() Javascript function calls. What’s going on here and how do we parse this stuff? To understand what this is about, we must go through a little journey across the technological landscape of the modern web. Note that I am not a frontend developer and don’t aspire to be. For the most part, I extract data from web apps and web sites, not develop them. The following is what I managed to understand from reading around.

So, it’s widely known that React is a very prominent frontend framework to develop web apps in JavaScript. Although one can develop web apps with the very basics - HTML, CSS and vanilla JavaScript, doing so is too simple for enterprise software teams and tech startups drunk on that sweet VC funding. They need more powerful abstractions, such as reusable components, UI state management, Virtual DOM to optimize performance by only redrawing parts of UI that changed and JSX syntax that extends Javascript language into Javascript-HTML hybrid dialect. More generally, React has emerged as a winner among many ZIRP-era Javascript frameworks by addressing the reactivity problem in web app development: how do we associate internal app state with visible UI stuff and how do we keep them in sync when one or the other changes.

Primarily React is only a frontend framework meant to develop code that runs in a browser. React alone cannot be used for full stack web development, as we also need something for the backend side of the app. To fill this gap, Next.js was developed. Next.js is another Javascript framework that works with React to provide features like request routing, performance optimization (e.g. making images smaller depending on the needs of frontend side) and Server Side Rendering (SSR). The latter will become important for our purposes in a moment. In addition to all of that, Next.js provides some tooling for easier deployment and nicer developer experience.

Those of us who work in web scraping have seen that many sites tend to have pages with <script id="__NEXT_DATA__" type="application/json"> tag that contains a large JSON document with all kinds of juicy data we can parse out of there. This is related to concept called hydration - embedding data within HTML document during server-side rendering so that it could be attached to React data model when bootstrapping all the React client-side machinery that makes the page interactive. For example, the first page of the product list is fetched with this __NEXT_DATA__ shebang, but if you go to the next page of the product list, REST API call will be initiated to fetch further data.

Screenshot 1

One simple way to browse this data is to type __NEXT_DATA__ into Console tab of browser dev tools panel. You will be presented with tree structure UI to explore it. An older post gives an example of parsing real data from __NEXT_DATA__ JSON document.

Screenshot 2 Screenshot 3

In some newer React/Next.js websites another approach is used to hydrate the pages that relies on something called React Server Components - a new feature of React that moves even more computation back to the backend side. If React Server Components is used, that involves splitting the initial data into chunks, serializing it to React-specific wire format and including it into the page through those self.__next_f.push() calls.

Screenshot 4

That may or may not make the pages smaller (see Next.js discussion #42170) and from a distance it looks like they’re making already complicated systems even more complicated. But as web scrapers developers we primarily care about extracting data from this mess. So how do we do that?

Serialised stuff in self.__next_f.push() calls is known as Next.js flight data. We need to parse it somehow. We could read through React.js/Next.js framework code and try to work it out, but fortunately someone already did the hard part for us. There’s njsparser Python module that we can use to scrape pages with Next.js flight data.

In njsparser project documentation we see the following Python snippet:

import requests
import njsparser
import json

# Here I get my page's html
response = requests.get("https://mediux.pro/user/r3draid3r04").text
# Then I parse it with njsparser
fd = njsparser.BeautifulFD(response)
# Then I iterate over the different classes `Data` in my flight data.
for data in fd.find_iter([njsparser.T.Data]):
    # Then I make sure that the content of my data is not None, and
    # check if the key `"user"` is in the data's content. If it is,
    # then i break the loop of searching.
    if data.content is not None and "user" in data.content:
        break
else:
    # If i didn't find it, i raise an error
    raise ValueError

# Now i have the data of my user
user = data.content["user"]
# And I can print the string i was searching for before
print(user["tagline"])

Let us extend this snippet a little to provide a way to input proxy URL with page URL to see what can be parsed out of the page:

import requests
import njsparser
from pprint import pprint

proxy_url = input("Proxy URL: ")

if proxy_url.strip() == "":
    proxy_url = None

proxies = {
    "https": proxy_url
}

url = input("Page URL: ")

response = requests.get(url, proxies=proxies, verify=False)
print(response.url, response.status_code)

fd = njsparser.BeautifulFD(response.text)
for data in fd.find_iter([njsparser.T.Data, njsparser.T.Element]):
    print(type(data))
    pprint(data)

Now, we can explore a real page with Next.js flight data:

$ python3 njsparser_text.py 
Proxy URL: 
Page URL: https://www.chilis.com/locations
/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/site-packages/urllib3/connectionpool.py:1099: InsecureRequestWarning: Unverified HTTPS request is being made to host 'www.chilis.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings
  warnings.warn(
https://www.chilis.com/locations?https= 200
Couldn't find an appropriate type for given class `None`. Giving `Element`.
Couldn't find an appropriate type for given class `None`. Giving `Element`.
<class 'njsparser.parser.types.Data'>
Data(value=['$',
            '$10',
            None,
            {'children': ['$', '$L11', None, {'promise': '$@12'}],
             'fallback': None}],
     value_class=None,
     index=7)
<class 'njsparser.parser.types.Element'>
Element(value={'digest': '$undefined',
               'error': None,
               'metadata': [['$',
                             'title',
                             '0',
                             {'children': "Chili's Grill & Bar"}],
                            ['$',
                             'meta',
                             '1',
                             {'content': "Chili's Grill & Bar. View menu, "
                                         'order online, and view restaurant '
                                         'hours.',
                              'name': 'description'}]]},
        value_class=None,
        index=18)
<class 'njsparser.parser.types.Element'>
Element(value={'digest': '$undefined',
               'error': None,
               'metadata': '$12:metadata'},
        value_class=None,
        index=12)
<class 'njsparser.parser.types.Data'>
Data(value=['$',
            '$L19',
            None,
            {'apiKey': '4967176e01a141828a5fad701f6faa79',
             'content': {'createdBy': 'g4MEiL4rhtSHs8OUcuU01J0Q8EN2',
                         'createdDate': 1712696242376,
                         'data': {'blocks': [{'@type': '@builder.io/sdk:Element',
                                              '@version': 2,
                                              'component': {'name': 'Image',
                                                            'options': {'altText': '404 '
                                                                                   'Error '
                                                                                   '- '
                                                                                   'Page '
                                                                                   'Not '
                                                                                   'Found. '
                                                                                   'Click '
                                                                                   'to '
                                                                                   'head '
                                                                                   'back '
                                                                                   'to '
                                                                                   'chilis.com '
                                                                                   'homepage. ',
                                                                        'aspectRatio': 0.563,
                                                                        'backgroundPosition': 'center',
                                                                        'backgroundSize': 'contain',
                                                                        'fitContent': True,
                                                                        'height': 594,
                                                                        'image': 'https://cdn.builder.io/api/v1/image/assets%2F4967176e01a141828a5fad701f6faa79%2Fe821f528372449eca0aaeb5e230dbcc5?width=2000',
                                                                        'lazy': False,
                                                                        'lockAspectRatio': False,
                                                                        'sizes': '(max-width: '
                                                                                 '638px) '
                                                                                 '100vw, '
                                                                                 '(max-width: '
                                                                                 '998px) '
                                                                                 '100vw, '
                                                                                 '143vw',
                                                                        'srcset': '$1a',
                                                                        'width': 1056}},
                                              'id': 'builder-30ef217134004c3aad5e4895a0017b5e',
                                              'linkUrl': '/',
                                              'properties': {'href': '/'},
                                              'responsiveStyles': {'large': {'boxSizing': 'border-box',
                                                                             'cursor': 'pointer',
                                                                             'display': 'flex',
                                                                             'flexDirection': 'column',
                                                                             'flexShrink': '0',
                                                                             'height': '100',
                                                                             'minHeight': '20px',
                                                                             'minWidth': '20px',
                                                                             'overflow': 'hidden',
                                                                             'pointerEvents': 'auto',
                                                                             'position': 'relative',
                                                                             'width': '100%'}},
                                              'tagName': 'a'},
                                             {'@type': '@builder.io/sdk:Element',
                                              'id': 'builder-pixel-vyr9xl1zoz',
                                              'properties': {'alt': '',
                                                             'aria-hidden': 'true',
                                                             'height': '0',
                                                             'role': 'presentation',
                                                             'src': 'https://cdn.builder.io/api/v1/pixel?apiKey=4967176e01a141828a5fad701f6faa79',
                                                             'width': '0'},
                                              'responsiveStyles': {'large': {'display': 'block',
                                                                             'height': '0',
                                                                             'opacity': '0',
                                                                             'overflow': 'hidden',
                                                                             'pointerEvents': 'none',
                                                                             'width': '0'}},
                                              'tagName': 'img'}],
                                  'state': {'deviceSize': 'large',
                                            'location': {'path': ['not-found'],
                                                         'pathname': '/not-found',
                                                         'query': {}}},
                                  'themeId': False,
                                  'title': '404 Error Page',
                                  'url': '/not-found'},
                         'firstPublished': 1712696453909,
                         'folders': [],
                         'id': 'a6f4d1a5049d47839e72c55aecda96cb',
                         'lastUpdated': 1712959355757,
                         'lastUpdatedBy': 'g4MEiL4rhtSHs8OUcuU01J0Q8EN2',
                         'meta': {'breakpoints': {'medium': 1023, 'small': 768},
                                  'hasLinks': True,
                                  'kind': 'page',
                                  'lastPreviewUrl': 'https://pp1.preview.chilis.com/not-found?builder.space=eecd8b66e2c94dd393b6f3fdd27468e7&builder.cachebust=true&builder.preview=page&builder.noCache=true&builder.allowTextEdit=true&__builder_editing__=true&builder.overrides.page=eecd8b66e2c94dd393b6f3fdd27468e7_a6f4d1a5049d47839e72c55aecda96cb&builder.overrides.eecd8b66e2c94dd393b6f3fdd27468e7_a6f4d1a5049d47839e72c55aecda96cb=eecd8b66e2c94dd393b6f3fdd27468e7_a6f4d1a5049d47839e72c55aecda96cb&builder.overrides.page:/not-found=eecd8b66e2c94dd393b6f3fdd27468e7_a6f4d1a5049d47839e72c55aecda96cb'},
                         'modelId': 'fc380e45ad4e4406a61d6c7dfb9d2479',
                         'name': '404 Error Page',
                         'published': 'published',
                         'query': [{'@type': '@builder.io/core:Query',
                                    'operator': 'is',
                                    'property': 'urlPath',
                                    'value': '/not-found'}],
                         'rev': 'z7bgev4he7i',
                         'screenshot': 'https://cdn.builder.io/api/v1/image/assets%2F4967176e01a141828a5fad701f6faa79%2Ffdcffe6f669b4e7586b2f75ab97c59fb',
                         'testRatio': 1,
                         'variations': {}}}],
     value_class=None,
     index=24)

Just like in the original form, we get the data in chunks. Each chunk is Python object with nested structure inside. Not much to explain here - just traverse the structure and extract the fields you need.

Lastly, njsparser project ships with a small CLI tool to check the pages if they have Next.js data to parse:

$ njsparser "https://www.chilis.com/locations"
Couldn't find an appropriate type for given class `None`. Giving `Element`.
Couldn't find an appropriate type for given class `None`. Giving `Element`.
Build Id: -KU-QA00B1JuhGcqPQ6eQ
The site contains flight data.
Pages:
- https://www.chilis.com/_app
- https://www.chilis.com/_error
$ njsparser "https://www.nike.com/t/air-jordan-1-retro-high-og-pro-green-womens-shoes-Hnf3kV/FD2596-101"
Build Id: ixvlnhp8VLcjaN1TTvfqm
The site contains a __NEXT_DATA__ script.
Pages:
- https://www.nike.com/_app
- https://www.nike.com/_error
- https://www.nike.com/t/[slug]
- https://www.nike.com/t/[slug]/[styleColor]

To conclude, we learned about Next.js flight data and how to use an open source Python library to make it tractable within web scrapers written in Python.

Setting up rooted Android emulator with Frida and mitmproxy

Fri, 12 Sep 2025 00:00:00 +0000

Previously I have covered how to set up mitmproxy with Android emulator or iOS device for the purposes of mobile app traffic interception. However, there is more to mobile app hacking, as not all apps allow their API calls to be hijacked by the simple setup I have described earlier. Some of the more secure apps implement a security mechanism called X.509 certificate pinning that entails checking if certificate on the server matches the expected one, thus requiring a more invasive approach to see intercept communications with backend systems. Furthermore, there is more to the app functionality than what is visible on the GUI and within traffic logs. Certain mobile apps implement proprietary HTTP headers to make adversarial API integrations more difficult to implement as they cannot be reverse engineered from intercepted API logs alone.

We will rely on following software, all free to download:

Android Studio - comprehensive IDE for Android app development that ships with some tooling we will need.
- ADB - CLI tool and communication interface to manage Android devices and emulators.
- avdmanager - CLI tool to manage Android emulators.
mitmproxy - MITM proxy server to intercept network traffic (primarily API calls over HTTPS).
Frida - dynamic instrumentation framework and toolkit. Think of it as more powerful equivalent of LD_PRELOAD, but mostly meant for mobile app reversing.
rootAVD - Android emulator rooting tool needed to get full access to system within emulator. This tool will also install Magisk - a set of tools for modding/customising Android.

Combination of mitmproxy and Frida will allow us to tamper with both communications and computations of the app, thus enabling a far deeper insight into inner workings than just sniffing API calls.

Broadly speaking, there are three kinds of Android emulators (also known as AVDs - Android Virtual Devices) that you can create through Android Studio tooling:

Google Play option comes with Play Store app, Chrome browser and several Google apps. This is the most restricted kind of emulator you can create, but it still can be rooted with rootAVD tool.
Google APIs option does not come with the Play Store app, but still contains some apps from Google and various client libraries needed to interact with Google services.
AOSP option is the purest form of Android system that comes from Android Open Source Project. Only the very basic web view app is available instead of full Chrome browser. However, this kind of emulator comes with root access.

Let us launch Virtual Device Manager from Android Studio by picking the option from context menu next to Clone Repository button. A window with list of emulators will appear. To create a new emulator, press the button with plus sign. Now we need to choose a hardware profile. A hardware profile is a set of characteristics (screen geometry, form factor, appearance and so on) that correspond to a physical device being emulated. Not only mobile phones and tablet can be emulated, but also smart TVs, automotive infotainment systems and XR headsets are supported. Depending on the option you choose, you may need to download additional system images and SDKs to get it running (in that case Android studio will walk you through the process).

Screenshot 1 Screenshot 2

For the sake of simplicity, let us pick Small Phone with Play Store and latest version of Android. We launch the emulator and let it boot.

Screenshot 3

Another way to create an emulator is to use avdmanager CLI tool. While this tool comes with Android Studio, it may not be loaded in your shell namespace by not being in one of the directories listed in PATH environment variable. I was able to rectify this on macOS by adding the following line to my ~/.zshrc file:

export PATH=$PATH:$HOME/Library/Android/sdk/cmdline-tools/latest/bin

Command for creating ARM-based small phone AVD is:

$ avdmanager create avd -n "SmallPhone" -d "small_phone" -g "google_apis_playstore" -k "system-images;android-36;google_apis_playstore;arm64-v8a"

This command contains specific hardware profile from output of avdmanager list devices in -d parameter, a system image path in -k parameter and AVD type in -g parameter.

Now, if we try to gain root access via ADB we will run into a restriction:

$ adb shell
emu64a:/ $ su
/system/bin/sh: su: inaccessible or not found

Now it is time to use rootAVD tool. If we launch this tool without any parameters, we will get some text explaining the usage and at the bottom there will some proposed commands we can run (with autodetected ramdisk file paths):

$ ./rootAVD
<...>

Command Examples:
./rootAVD.sh
./rootAVD.sh ListAllAVDs
./rootAVD.sh InstallApps

./rootAVD.sh system-images/android-34/google-xr/arm64-v8a/ramdisk.img
./rootAVD.sh system-images/android-34/google-xr/arm64-v8a/ramdisk.img FAKEBOOTIMG
./rootAVD.sh system-images/android-34/google-xr/arm64-v8a/ramdisk.img DEBUG PATCHFSTAB GetUSBHPmodZ
./rootAVD.sh system-images/android-34/google-xr/arm64-v8a/ramdisk.img restore
./rootAVD.sh system-images/android-34/google-xr/arm64-v8a/ramdisk.img InstallKernelModules
./rootAVD.sh system-images/android-34/google-xr/arm64-v8a/ramdisk.img InstallPrebuiltKernelModules
./rootAVD.sh system-images/android-34/google-xr/arm64-v8a/ramdisk.img InstallPrebuiltKernelModules GetUSBHPmodZ PATCHFSTAB DEBUG
./rootAVD.sh system-images/android-34/google-xr/arm64-v8a/ramdisk.img AddRCscripts

The file in question is crucial for Android boot process, as it contains a small filesystem needed to bootstrap rest of Android. In this case Small Phone AVD does not correspond to ramdisk listed, but we can get more options by launching ./rootAVD.sh ListAllAVDs command, which leads us to the following command:

$ ./rootAVD.sh system-images/android-36/google_apis_playstore/arm64-v8a/ramdisk.img

At one point it will ask to pick Magisk version. It is fine to pick the default one by pressing Enter. Once done, it will shut down the emulator. We will need to force it to go through the entire boot process by picking Cold Boot from additional menu in Virtual Device Manager window. You will see that Magisk app is available on the virtual phone. It will require some additional setup and one more reboot of the emulator.

Screenshot 4

Now, if we try gaining root access via ADB again, Magisk GUI will ask us to approve it for the first time, but we will be able to get it:

$ adb shell
emu64a:/ $ su
emu64a:/ # whoami
root

Screenshot 5

Now we can launch mitmproxy and go through procedure described in my older post up until the point where we need to trust a certificate. On modern Android we need to go to Settings app, search for “CA certificate” and go through the simple flow to install and trust the certificate downloaded from MITM proxy server. Once that is done, we can go to some random website via Google Chrome and verify that HTTP traffic can be inspected via mitmproxy.

Screenshot 6 Screenshot 7 Screenshot 8

So it’s time to install Frida. There are two parts to Frida: 1) Python tooling to be launched in the host system and 2) binary server running under root user within Android system. Architecture of binary part has to match your emulator architecture (ARM or x86).

Python part of Frida can be install via PIP:

$ pip3 install frida-tools

Let us download Frida server binary, decompress it and rename the binary file for our convenience.

$ wget https://github.com/frida/frida/releases/download/17.2.17/frida-server-17.2.17-android-arm64.xz
$ unxz frida-server-17.2.17-android-arm64.xz 
$ mv frida-server-17.2.17-android-arm64 frida-server

Now we can push the Frida server via ADB and launch it under root (keep this running in terminal session):

$ adb push frida-server /data/local/tmp
frida-server: 1 file pushed, 0 skipped. 226.4 MB/s (52019048 bytes in 0.219s)
$ adb shell
emu64a:/ $ su
emu64a:/ # chmod +x /data/local/tmp/frida-server                               
emu64a:/ # /data/local/tmp/frida-server

One of the Android apps that implement TLS cert pinning is Hollister Co. app. You can download the APK file from third party source and install it by dragging the APK file into emulator window. Now everything comes together. We are going to use generalised Bypass SSL Pinning script that covers a variety of ways certificate pinning can be implemented and attempts to disable them.

$ frida -U --codeshare Q0120S/bypass-ssl-pinning -f com.abercrombie.hollister
     ____
    / _  |   Frida 17.2.16 - A world-class dynamic instrumentation toolkit
   | (_| |
    > _  |   Commands:
   /_/ |_|       help      -> Displays the help system
   . . . .       object?   -> Display information about 'object'
   . . . .       exit/quit -> Exit
   . . . .
   . . . .   More info at https://frida.re/docs/home/
   . . . .
   . . . .   Connected to Android Emulator 5554 (id=emulator-5554)
Spawned `com.abercrombie.hollister`. Resuming main thread!              
[Android Emulator 5554::com.abercrombie.hollister ]-> ---
Unpinning Android app...
[+] SSLPeerUnverifiedException auto-patcher
[+] HttpsURLConnection (setDefaultHostnameVerifier)
[+] HttpsURLConnection (setSSLSocketFactory)
[+] HttpsURLConnection (setHostnameVerifier)
[+] SSLContext
[+] TrustManagerImpl
[ ] OkHTTPv3 (list)
[ ] OkHTTPv3 (cert)
[ ] OkHTTPv3 (cert array)
[ ] OkHTTPv3 ($okhttp)
[ ] Trustkit OkHostnameVerifier(SSLSession)
[ ] Trustkit OkHostnameVerifier(cert)
[ ] Trustkit PinningTrustManager
[ ] Appcelerator PinningTrustManager
[ ] OpenSSLSocketImpl Conscrypt
[ ] OpenSSLEngineSocketImpl Conscrypt
[ ] OpenSSLSocketImpl Apache Harmony
[ ] PhoneGap sslCertificateChecker
[ ] IBM MobileFirst pinTrustedCertificatePublicKey (string)
[ ] IBM MobileFirst pinTrustedCertificatePublicKey (string array)
[ ] IBM WorkLight HostNameVerifierWithCertificatePinning (SSLSocket)
[ ] IBM WorkLight HostNameVerifierWithCertificatePinning (cert)
[ ] IBM WorkLight HostNameVerifierWithCertificatePinning (string string)
[ ] IBM WorkLight HostNameVerifierWithCertificatePinning (SSLSession)
[ ] Conscrypt CertPinManager
[ ] CWAC-Netsecurity CertPinManager
[ ] Worklight Androidgap WLCertificatePinningPlugin
[ ] Netty FingerprintTrustManagerFactory
[ ] Squareup CertificatePinner (cert)
[ ] Squareup CertificatePinner (list)
[ ] Squareup OkHostnameVerifier (cert)
[ ] Squareup OkHostnameVerifier (SSLSession)
[+] Android WebViewClient (SslErrorHandler)
[ ] Android WebViewClient (WebResourceError)
[ ] Apache Cordova WebViewClient
[ ] Boye AbstractVerifier
[ ] Appmattus (CertificateTransparencyInterceptor)
[ ] Appmattus (CertificateTransparencyTrustManager)
Unpinning setup completed
---
  --> Bypassing Trustmanager (Android < 7) request
  --> Bypassing TrustManagerImpl checkTrusted 
  --> Bypassing TrustManagerImpl checkTrusted 
  --> Bypassing TrustManagerImpl checkTrusted 
  --> Bypassing Trustmanager (Android < 7) request
  --> Bypassing TrustManagerImpl checkTrusted 
  --> Bypassing Trustmanager (Android < 7) request
  --> Bypassing Trustmanager (Android < 7) request
  --> Bypassing TrustManagerImpl checkTrusted 
  --> Bypassing TrustManagerImpl checkTrusted 
  --> Bypassing Trustmanager (Android < 7) request
  --> Bypassing Trustmanager (Android < 7) request
  --> Bypassing Trustmanager (Android < 7) request
  --> Bypassing Trustmanager (Android < 7) request
  --> Bypassing Trustmanager (Android < 7) request
  --> Bypassing Trustmanager (Android < 7) request
  --> Bypassing TrustManagerImpl checkTrusted 
  --> Bypassing Trustmanager (Android < 7) request
  --> Bypassing Trustmanager (Android < 7) request
  --> Bypassing TrustManagerImpl checkTrusted 
  --> Bypassing Trustmanager (Android < 7) request
  --> Bypassing TrustManagerImpl checkTrusted 
  --> Bypassing TrustManagerImpl checkTrusted 
  --> Bypassing TrustManagerImpl checkTrusted 
  --> Bypassing TrustManagerImpl checkTrusted 
  --> Bypassing TrustManagerImpl checkTrusted 
  --> Bypassing TrustManagerImpl checkTrusted

Hollister Co. app now can be operated successfully and we can also see the HTTP flows being intercepted in mitmproxy.

Screenshot 9

If we look through the script we just used, we see some JavaScript code with many try-catch blocks, for example:

        // OpenSSLSocketImpl Conscrypt
        try {
            const OpenSSLSocketImpl = Java.use('com.android.org.conscrypt.OpenSSLSocketImpl');
            OpenSSLSocketImpl.verifyCertificateChain.implementation = function (certRefs, JavaObject, authMethod) {
                console.log('  --> Bypassing OpenSSLSocketImpl Conscrypt');
            };
            console.log('[+] OpenSSLSocketImpl Conscrypt');
        } catch (err) {
            console.log('[ ] OpenSSLSocketImpl Conscrypt');
        }

We can see that this piece of code attacks one of the possible cert pinning implementations by replacing (at runtime!) the exact code to verify certificate chain with small function that only prints a debug log. This is an example of the dynamic instrumentation concept that Frida project centers around.

jq: tool and language for JSON processing

Sat, 14 Jun 2025 00:00:00 +0000

JSON is the dominant data representation format for most of the modern RESTful APIs and certain automation/devops tools (Terraform, AWS CLI, kubectl, etc.) optionally output data in JSON format to make it machine readable. Some datasets (e.g. service price lists from health insurance companies) are available primarily as JSON files. For these reasons parsing, generation, modification and analysis of JSON documents is nothing new for many explorers of cyberspace. Indeed, there are libraries and tools to wrangle JSONified data in pretty much all popular general purpose programming languages. But what about shell scripts? When one is programming in Bash or Zsh it might be necessary to run some curl(1) snippet and extract parts of API response, but your standard Unix text processing tools - awk(1), sed(1), grep(1) and others largely predate JSON popularity and are not designed with nested data trees in mind. jq is CLI tool that was designed to fill the gap here and addresses the need of making JSON wrangling easier within the Unix/Linux environment. But it is not merely a CLI tool. Like AWK, it is also a Turing-complete domain specific language that lets you implement arbitrarily complex (within limits of computability) JSON document processing scripts.

One can install jq via the usual package managers:

brew install jq on macOS with Homebrew.
apt-get install jq on Debian and many Debian-based distributions.
pacman -S jq on Arch Linux.
pkg install jq on FreeBSD.

Furthermore, jq project ships ready-to-use binary files as part of their release deliverables. To get the official Docker image with jq one can run docker pull ghcr.io/jqlang/jq:latest.

There is also jqplay.org - a web playground for playing with jq language without installing anything (but not for processing large JSON documents).

The syntax of jq has similarities with JSON syntax. The very simplest thing one can do in jq is pretty printing the input for readability (this can be turned off with --compact-output or -c):

$ echo '{"x": 1, "y": 2}' | jq '.'
{
  "x": 1,
  "y": 2
}

By default jq makes output colored for easier reading. We used jq identity statement as the first argument.

We may want to create JSON object or list in jq language. Syntax to do so is very similar to JSON notation:

$ jq --null-input '{a: 1, b: 2}'
{
  "a": 1,
  "b": 2
}
$ jq --null-input '[1,3,5,7,11]'
[
  1,
  3,
  5,
  7,
  11
]

To extract a single field from JSON object we can use object identifier-index syntax - dot character with field name:

$ echo '{"x": 1, "y": 2}' | jq '.x'
1

This can be done across multiple levels:

$ echo '{"coord": {"x": 1, "y": 2}}' | jq '.coord.y'
2

Like other programming languages, JSON arrays can be indexed:

$ echo '{"coords": [{"x": 1.1, "y": 2.5}]}' | jq '.coords[0].y'
2.5

Like in Python, arrays can be sliced by index range:

$ echo '[1,2,3,4,5]' | jq '.[1:3]'
[
  2,
  3
]

If used without indices, [] is iterator - it returns all values within the input list or object:

$ echo '[1,2,3,4,5]' | jq '.[]'   
1
2
3
4
5
$ echo '{"a": 1, "b": 2}' | jq '.[]'
1
2

Note that first index in the range is inclusive, but not the second one - second index tells jq to extract values up to that index.

JSON data types have the same syntax in jq language as in JSON notation:

$ jq --monochrome-output --null-input 'null'   
null
$ jq --null-input '1.2'   
1.2
$ jq --null-input '"str"' 
"str"
$ jq --null-input 'false'
false
$ jq --null-input '{"latitude": 1.1, "longitude": 2.2}' 
{
  "latitude": 1.1,
  "longitude": 2.2
}
$ jq --null-input '["a", "b", "c"]' 
[
  "a",
  "b",
  "c"
]

Like in Unix environment, pipe character can be used to build pipelines of operations:

$ jq --null-input '{"latitude": 1.1, "longitude": 2.2} | .latitude'
1.1

Comma character can be used to concatenate outputs from two or more operations into single stream:

$ jq --null-input '{"latitude": 1.1, "longitude": 2.2} | .latitude, .longitude'  
1.1
2.2

Furthermore, jq language allows using question mark character as optional operator. Like in Swift programming language, this allows for parts of input (e.g. a subtree in JSON document) to be missing without crashing the program. If something is not missing, further operations will be executed. If it is missing, the entire chain of operations will return null value:

$ echo '{"coords": [{"x": 1.1, "y": 2.5}]}' | jq '.coords?[0].y'
2.5
$ echo '{"coords": [{"x": 1.1, "y": 2.5}]}' | jq '.coordinates?[0].y'
null

Like all the proper programming languages, jq has operators for basic math operations:

$ jq --null-input '1 + 1'
2
$ jq --null-input '1 - 1'
0
$ jq --null-input '1 * 2'
2
$ jq --null-input '1 / 2'
0.5
$ jq --null-input '5 % 2' 
1

Multiplication operator can also be used to make repeated string:

$ jq --null-input '"cyber" * 3'
"cybercybercyber"

Furthermore, jq has some built-in functions for common computations and data processing operations. For example, length is equivalent to Python’s len() function:

$ echo '"abc"' | jq 'length'   
3
$ echo '[1, 2, 3, 4]' | jq 'length'
4

The jq language does not have for or while loops. Instead, we are supposed to use two brackets ([]) - iteration operator and/or range function:

$ jq --null-input 'range(100)'
0
1
2
3
4
...
96
97
98
99
100
$ jq --null-input '[range(100)]' | jq '.[] | . + 1000' | head
1000
1001
1002
1003
1004
1005
1006
1007
1008
1009

In jq language map is equivalent to map() in Python and select is equivalent to filter().

This allows us to implement data processing in FP-like manner:

$ echo "[1000, 2000, 3000, 5000, 9001, 10240]" | jq '.[] | select(. > 9000)'
9001
10240
$ echo '["corpothieves", "must", "die"]' | jq 'map(length)'
[
  12,
  4,
  3
]

If we don’t want to use select we can use regular if-then-else control flow feature of jq:

$ echo "9001" | jq 'if . > 9000 then "over nine thousand" else "below 9000" end'
"over nine thousand"

I asked Google Gemini 2.5 to write Fizzbuzz code in jq language. Take a look:

range(1; 101) |
  if . % 15 == 0 then
    "FizzBuzz"
  elif . % 3 == 0 then
    "Fizz"
  elif . % 5 == 0 then
    "Buzz"
  else
    .
  end

It can be launched with following command:

$ jq -n -r -f fizzbuzz_standard.jq

There is also IEEE754 double precision floating point number support so you can do trigonometry and stuff. See the documentation for a list of built-in functions available.

Many programming languages provide a way to develop reusable pieces of code that can be accessed via API. We already tried some jq functions that are shipped with standard installation. To define jq module we must create file with .jq extension and write one more jq functions in there like it is done in src/builtin.jq file.

Now it’s time for a real-worldish example of using jq for data extraction. But first, let me show you a trick. Pretty much all Shopify stores that don’t use custom frontend expose some APIs with product data. First, /products.json endpoint provides a product list. It acts as paginated Product List Page (PLP) API with parameters page for page number (starting with 1) and limit for page size (can be up to 250). See the following examples:

It should be noted that sometimes primary store domain cannot be used and should be replaced with .myshopify subdomain that can be found in page source. For our example store that would be online-hypebeastbaltics.myshopify.com.

Screenshot 1

If you want individual product data from the store that can be retrieved by appending .js or .json to PDP URL path. This will give you slightly different versions of the data and, depending on the target, fields regarding inventory or product availability might be present in one form, but not in another. For example the following URLs can be used for API scraping:

It is also possible to retrieve products from collection, e.g.:

This can be used to gather data programmatically at PLP and PDP levels from Shopify store with little to no need for HTML page scraping. Since the response data format is JSON, we can use jq to extract the interesting parts. We will still need something else to implement the remaining parts of API scraping code. In this case we will use Bash scripting language.

#!/bin/bash

set -x

URL="https://hypebeastbaltics.com/products.json"
PER_PAGE=250
PAGE=1

echo "id,title,body_html,vendor,product_type,price,handle" > hbb.csv

while true ; do
    JSON_STR=$(curl "$URL?page=$PAGE&limit=$PER_PAGE")
    echo "$JSON_STR" | jq -r '.products[] | [.id, .title, .body_html, .vendor, .product_type, .variants[0].price, .handle] | @csv' >> hbb.csv
    N_PRODUCTS=$(echo "$JSON_STR" | jq -r '.products | length')
    if [[ "$N_PRODUCTS" -lt "$PER_PAGE" ]]; then
        break
    fi
    PAGE=$((PAGE+1))
done

The jq codebase is in C, but the underlying “backend” of the language can be used as a shared library - libjq. This would enable using it as DSL outside shell scripts. But the thing is, unless what you do is in the realm of systems programming you probably do not want to use libjq C API in your own code. For our convenience there are wrappers in higher level languages:

For example, the Python module makes API scraping code less tedious:

>>> import requests
>>> import jq
>>> from pprint import pprint
>>> resp = requests.get("https://hypebeastbaltics.com/products.json")
>>> titles = jq.compile(".products[].title").input(text=resp.text).all()
>>> pprint(titles)
['Denver Broncos fitted hat brown corduroy 7 3/4',
 'Denver Broncos fitted hat Black 8',
 'Dallas Cowboys Superbowl fitted hat navy 7 1/8',
 'Minnesota Vikings fitted hat pink 7 1/4',
 'San Diego Padres fitted hat white green 7 1/2',
 'San Francisco 49ers fitted hat  7 3/8',
 'Los Angeles Rams fitted hat white 7 3/8',
 'San Antonio Spurs fitted hat Grey 7 3/8',
 'New York Yankees fitted hat Navy 6 3/8',
 'CDG Polka Dot With Upside Down Heart T-Shirt White',
 'Used adidas Yeezy Boost 350 V2 Cream',
 'Used Adidas Ultra Boost 4.0 Bape Camo',
 'Used Adidas Yeezy Boost 350 V2 Earth',
 'Used Yeezy 500 Utility Black',
 'Used adidas Yeezy Boost 350 V2 MX Oat',
 'Used Yeezy 700 V3 Clay Brown',
 'Used Yves Saint Laurent Leather high trainers',
 'Used Louis Vuitton Boxing High Top Triple Black',
 'Pop Mart Labubu The Monsters Lazy Yoga Series Figures Blind Box',
 'Pop Mart The Monsters Labubu Zimomo Angel in Clouds Figure (No Tote)',
 'Nike x Jacquemus Le Polo Off-White',
 'Nike x NOCTA NRG Knit Long Sleeve Top Cobalt Bliss/Dark Obsidian',
 'COMING 6TH JUNE Pop Mart Labubu The Monsters Have a Seat Vinyl Plush Sealed '
 'Case (6 Blind Boxes)',
 'COMING 6TH JUNE Pop Mart Labubu The Monsters Tasty Macarons Vinyl Plush '
 'Pendant Single Blind Box',
 'Corteiz Olympic Shuku Nigeria Jacket Green/White',
 'Nike x NOCTA NRG Short Cobalt Bliss/White',
 'Pop Mart Labubu The Monsters Big into Energy Series Vinyl Plush Pendant '
 'Sealed Case (6 Blind Boxes)',
 'COMING 6TH JUNE Pop Mart Labubu The Monsters Exciting Macarons Vinyl Face '
 'Blind Box Whole Set',
 'Used Supreme Bandana Tarp Side Bag',
 'Used Carolina Panthers Fitted Cap']

Gojq project reimplements the language in pure Go and does not rely on the C codebase. It can be used as CLI tool and integrated into other software as library.

Some interesting codebases using jq language:

jnv - interactive tool to browse JSON data and prototype jq one-liners in a terminal.
jqview - Qt-based GUI app based on Gojq.
tq - tool to manipulate Terraform objects with jq language.

Katana: web crawler for offensive security and web exploration

Tue, 01 Oct 2024 00:00:00 +0000

Katana is CLI tool and Go library for automatically traversing (crawling) across all pages of given website(s) to map them out. It can work in two main modes - requests-based and through browser automation (headful or headless). To allow for discovery of API endpoints it can optionally do JavaScript parsing even when running in requests-based mode. Furthermore, Katana can do passive crawling by leveraging pre-crawled data from Internet Archive Wayback Machine, CommonCrawl and Alien Vault OTX. Since mapping out site pages and APIs is useful for security research activities (e.g bug bounty hunting) Katana is designed to fit into larger automation workflows, esp. when used together with other tooling from Project Discovery.

On macOS Katana is available through Homebrew and there is also official Docker image on Docker Hub that includes a Chromium browser.

If you have Golang development environment you can install it via Go CLI:

$ go install github.com/projectdiscovery/katana/cmd/katana@latest

If we simply want to see a list of pages a site has we only have to pass the site URL via -u parameter:

$ katana -u https://public-firing-range.appspot.com/

   __        __                
  / /_____ _/ /____ ____  ___ _
 /  '_/ _  / __/ _  / _ \/ _  /
/_/\_\\_,_/\__/\_,_/_//_/\_,_/							 

		projectdiscovery.io

[INF] Current katana version v1.1.0 (latest)
[INF] Started standard crawling for => https://public-firing-range.appspot.com/
https://public-firing-range.appspot.com/
https://public-firing-range.appspot.com/vulnerablelibraries/index.html
https://public-firing-range.appspot.com/stricttransportsecurity/index.html
https://public-firing-range.appspot.com/urldom/index.html
https://public-firing-range.appspot.com/reverseclickjacking/
https://public-firing-range.appspot.com/reflected/index.html
https://public-firing-range.appspot.com/mixedcontent/index.html
https://public-firing-range.appspot.com/tags/index.html
https://public-firing-range.appspot.com/redirect/index.html
https://public-firing-range.appspot.com/remoteinclude/index.html
https://public-firing-range.appspot.com/insecurethirdpartyscripts/index.html
https://public-firing-range.appspot.com/leakedcookie/index.html
https://public-firing-range.appspot.com/flashinjection/index.html
https://public-firing-range.appspot.com/vulnerablelibraries/jquery.html
...

But now we don’t see status codes and we’re not even sure if all of these pages load properly. We can use -mdc argument with Project Discovery DSL snippet to filter out only responses with status code 200:

$ katana -u https://public-firing-range.appspot.com/ -mdc 'status_code == 200'

We can use this tool to store responses in case you want to separate web crawling from HTTP parsing:

$ katana -u https://books.toscrape.com/ -store-response

This decouples former from the latter so that you can download all the pages once and then work on code that parses the HTML documents. The above command creates a new directory named katana_responses with an index.txt file listing requests, status codes and relative file paths:

$ head index.txt 
katana_response/books.toscrape.com/3060d727707e12334c7134afe04df480af3f0fdf.txt https://books.toscrape.com/ (200 OK)
katana_response/books.toscrape.com/f77b190299d9ad01298054c2b1ce13f35765e4d1.txt https://books.toscrape.com/static/oscar/js/bootstrap-datetimepicker/locales/bootstrap-datetimepicker.all.js (200 OK)
katana_response/books.toscrape.com/9e4c7d8e5630912730d1194fec599ec3f5f8580e.txt https://books.toscrape.com/static/oscar/js/bootstrap-datetimepicker/bootstrap-datetimepicker.js (200 OK)
katana_response/books.toscrape.com/7fdd4edbfcbe5d86df9904b18b6747e7cf7f0efa.txt https://books.toscrape.com/static/oscar/js/oscar/ui.js (200 OK)
katana_response/books.toscrape.com/cd148dbefc26230ec0d29104eb0408ae640720d2.txt https://books.toscrape.com/static/oscar/js/bootstrap3/bootstrap.min.js (200 OK)
katana_response/books.toscrape.com/0eb507686604bb3002bdf01dc8d1b917c7352867.txt https://books.toscrape.com/static/oscar/css/datetimepicker.css (200 OK)
katana_response/books.toscrape.com/0a63ac4de25fb4ba06a76137240cc663a716b506.txt https://books.toscrape.com/static/oscar/js/bootstrap-datetimepicker/bootstrap-datetimepicker.css (200 OK)
katana_response/books.toscrape.com/a2652354fa625fb0f225b7cd87f035f8f6d6af2c.txt https://books.toscrape.com/catalogue/its-only-the-himalayas_981/index.html (200 OK)
katana_response/books.toscrape.com/bf2072cbc394277fe024f4992dde7a498b7d6a9f.txt https://books.toscrape.com/catalogue/libertarianism-for-beginners_982/index.html (200 OK)
katana_response/books.toscrape.com/e4f9cda62c7f551a9e54439e5ed27f6e3306902a.txt https://books.toscrape.com/catalogue/mesaerion-the-best-science-fiction-stories-1800-1849_983/index.html (200 OK)

Each of the page snapshot files retain original URL, request/response headers and response payload:

$ head -n 50 books.toscrape.com/f77b190299d9ad01298054c2b1ce13f35765e4d1.txt 
https://books.toscrape.com/static/oscar/js/bootstrap-datetimepicker/locales/bootstrap-datetimepicker.all.js


GET /static/oscar/js/bootstrap-datetimepicker/locales/bootstrap-datetimepicker.all.js HTTP/1.1
Host: books.toscrape.com
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 11_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36
Accept-Encoding: gzip



HTTP/1.1 200 OK
Content-Length: 31995
Accept-Ranges: bytes
Connection: keep-alive
Content-Type: application/javascript
Date: Tue, 20 Aug 2024 09:17:54 GMT
Etag: "63e40de9-7cfb"
Last-Modified: Wed, 08 Feb 2023 21:02:33 GMT
Strict-Transport-Security: max-age=0; includeSubDomains; preload

/**
* Arabic translation for bootstrap-datetimepicker
* Ala' Mohammad <amohammad@birzeit.ecu>
*/
;(function($){
	$.fn.datetimepicker.dates['ar'] = {
		days: ["الأحد", "الاثنين", "الثلاثاء", "الأربعاء", "الخميس", "الجمعة", "السبت", "الأحد"],
		daysShort: ["أحد", "اثنين", "ثلاثاء", "أربعاء", "خميس", "جمعة", "سبت", "أحد"],
		daysMin: ["أح", "إث", "ث", "أر", "خ", "ج", "س", "أح"],
		months: ["يناير", "فبراير", "مارس", "أبريل", "مايو", "يونيو", "يوليو", "أغسطس", "سبتمبر", "أكتوبر", "نوفمبر", "ديسمبر"],
		monthsShort: ["يناير", "فبراير", "مارس", "أبريل", "مايو", "يونيو", "يوليو", "أغسطس", "سبتمبر", "أكتوبر", "نوفمبر", "ديسمبر"],
		today: "هذا اليوم",
		suffix: [],
		meridiem: [],
		rtl: true
	};
}(jQuery));
/**
 * Bulgarian translation for bootstrap-datetimepicker
 * Apostol Apostolov <apostol.s.apostolov@gmail.com>
 */
;(function($){
	$.fn.datetimepicker.dates['bg'] = {
		days: ["Неделя", "Понеделник", "Вторник", "Сряда", "Четвъртък", "Петък", "Събота", "Неделя"],
		daysShort: ["Нед", "Пон", "Вто", "Сря", "Чет", "Пет", "Съб", "Нед"],
		daysMin: ["Н", "П", "В", "С", "Ч", "П", "С", "Н"],
		months: ["Януари", "Февруари", "Март", "Април", "Май", "Юни", "Юли", "Август", "Септември", "Октомври", "Ноември", "Декември"],
		monthsShort: ["Ян", "Фев", "Мар", "Апр", "Май", "Юни", "Юли", "Авг", "Сеп", "Окт", "Ное", "Дек"],
		today: "днес",
		suffix: [],

Katana is not primarily designed to extract information from web pages - it is crawling, not scraping tool. For those in need of a comprehensive web scraping framework Colly and Scrapy are well established options. However, it is is possible to do regex-based data extraction by putting regular expressions in YAML file like the following example from the README.md file (simplified for brevity):

- name: email
  type: regex
  regex:
  - '([a-zA-Z0-9._-]+@[a-zA-Z0-9._-]+\.[a-zA-Z0-9_-]+)'

Once we have fields configured, we can run Katana to extract some data:

$ katana -u https://allbirds.com -f email -flc field-config.yaml 


   __        __                
  / /_____ _/ /____ ____  ___ _
 /  '_/ _  / __/ _  / _ \/ _  /
/_/\_\\_,_/\__/\_,_/_//_/\_,_/							 

		projectdiscovery.io

[INF] Current katana version v1.1.0 (latest)
[INF] Started standard crawling for => https://allbirds.com
help@allbirds.com
help@allbirds.com
help@allbirds.com
help@allbirds.com
help@allbirds.com
help@allbirds.com
help@allbirds.com
help@allbirds.com
privacy@allbirds.com
privacy@allbirds.com
privacy@allbirds.com
privacy@allbirds.com
privacy@allbirds.com
...

This enables Katana to be used for opportunistic contact data (email and/or phone number) scraping across a list of websites (which can be passed to Katana one per line via standard input). For example, one could source a list of companies within a niche by Google dorking or scraping a business directory website, then run Katana with the above custom field configuration to traverse the list of websites and grab emails whenever they happen to be somewhere on the page. Contacts most likely will be generic to the entire company, but this is significantly cheaper than running the domain list through enrichment API. Some growth hackers are using Scrapebox, a proprietary Windows desktop program to do this, but Katana requires no GUI environment if you want to run it on the server and is 100% open source.

However, regular expressions should be used sparingly for web scraping and one should refrain from trying to parse HTML with them. Consider that an anti-pattern.

If you run Katana with -js-crawl CLI argument it will download JavaScript files and parse them to find API endpoints:

$ katana -u https://allbirds.com -js-crawl

   __        __                
  / /_____ _/ /____ ____  ___ _
 /  '_/ _  / __/ _  / _ \/ _  /
/_/\_\\_,_/\__/\_,_/_//_/\_,_/							 

		projectdiscovery.io

[INF] Current katana version v1.1.0 (latest)
[INF] Started standard crawling for => https://allbirds.com
https://allbirds.com
https://www.allbirds.com/js.iterable.com/analytics.js
https://www.allbirds.com/cdn/shop/t/2473/assets/homepage.a1ce3f2b8cdbde6e27e6.chunk.js
https://www.allbirds.com/cdn/shop/t/2473/assets/44.1560197b842f08b3fe77.chunk.js
https://www.allbirds.com/cdn/shop/t/2473/assets/34.390ec96b64c918e7a51f.chunk.js
https://www.allbirds.com/cdn/shop/t/2473/assets/33.f428389d95947b0ea31f.chunk.js
https://www.allbirds.com/cdn/shop/t/2473/assets/32.c205e74564192b41988c.chunk.js
https://www.allbirds.com/cdn/shop/t/2473/assets/35.aec7b6810f22105211fb.chunk.js
https://www.allbirds.com/cdn/shop/t/2473/assets/43.8f8d6489f56adf20dcff.chunk.js
https://www.allbirds.com/cdn/shopifycloud/shopify/assets/shop_events_listener-61fa9e0a912c675e178777d2b27f6cbd482f8912a6b0aa31fa3515985a8cd626.js
https://www.allbirds.com/cdn/shop/t/2473/assets/29.81f74e37a33dc7fbe9dd.chunk.js
https://www.allbirds.com/cdn/shop/t/2473/assets/42.b0afa4f09987aaabeeaf.chunk.js
https://www.allbirds.com/filter.json
https://www.allbirds.com/reviews.json
https://www.allbirds.com/cdn/s/trekkie.storefront.d0db9c6b604f2af4af0875dc118feaf816931b65.min.js
https://www.allbirds.com/graphql.json
...

To take it a bit further and potentially find more APIs one may add -jsluice argument that enables JS parsing with jsluice library (at the cost of increased RAM consumption).

This may be useful when exploring a website for API scraping or security assessment purposes.

By default Katana prints results to standard output. You can use -silent option to disable the ASCII art logo or -o to provide a file the output is to be written. Furthermore, -jsonl option formats each line into JSONL format that can be further processed programmatically.

When used from Go code, Katana is used like framework, not a library - you have to configure it to run the crawl for you and it will keep calling the callback function you provide on each page it goes through. See the following snippet that was made by extending the sample code Katana project provides:

package main

import (
	"bytes"
	"log"
	"math"
	"strconv"

	"github.com/goccy/go-graphviz"
	"github.com/goccy/go-graphviz/cgraph"
	"github.com/projectdiscovery/gologger"
	"github.com/projectdiscovery/katana/pkg/engine/standard"
	"github.com/projectdiscovery/katana/pkg/output"
	"github.com/projectdiscovery/katana/pkg/types"
)

func main() {
	g := graphviz.New()
	graph, _ := g.Graph()
	defer func() {
		graph.Close()
		g.Close()
	}()

	nodes := map[string]*cgraph.Node{}

	options := &types.Options{
		MaxDepth:     3,               // Maximum depth to crawl
		FieldScope:   "rdn",           // Crawling Scope Field
		BodyReadSize: math.MaxInt,     // Maximum response size to read
		Timeout:      10,              // Timeout is the time to wait for request in seconds
		Concurrency:  10,              // Concurrency is the number of concurrent crawling goroutines
		Parallelism:  10,              // Parallelism is the number of urls processing goroutines
		Delay:        0,               // Delay is the delay between each crawl requests in seconds
		RateLimit:    150,             // Maximum requests to send per second
		Strategy:     "breadth-first", // Visit strategy (depth-first, breadth-first)
		OnResult: func(result output.Result) { // Callback function to execute for result
			referer := result.Request.Source

			if referer != "" {
				gologger.Info().Msg(referer + " -> " + result.Request.URL)
			} else {
				gologger.Info().Msg(result.Request.URL)
			}

			node, _ := graph.CreateNode(result.Request.URL)
			nodes[result.Request.URL] = node
			prevNode := nodes[referer]

			if prevNode != nil {
				e, _ := graph.CreateEdge("", prevNode, node)
				e.SetLabel(strconv.Itoa(result.Response.StatusCode))
			}
		},
	}
	crawlerOptions, err := types.NewCrawlerOptions(options)
	if err != nil {
		gologger.Fatal().Msg(err.Error())
	}
	defer crawlerOptions.Close()
	crawler, err := standard.New(crawlerOptions)
	if err != nil {
		gologger.Fatal().Msg(err.Error())
	}
	defer crawler.Close()
	var input = "https://public-firing-range.appspot.com/"
	err = crawler.Crawl(input)
	if err != nil {
		gologger.Warning().Msgf("Could not crawl %s: %s", input, err.Error())
	}

	var buf bytes.Buffer
	if err := g.Render(graph, "dot", &buf); err != nil {
		log.Fatal(err)
	}

	if err := g.RenderFilename(graph, graphviz.SVG, "katana.svg"); err != nil {
		log.Fatal(err)
	}

}

Here we use Graphviz wrapper library to keep track of what links were followed between pages as crawling was in progress and render the resulting graph into SVG file. The end result of this is admittedly not very interesting, but the above code can be viewed as example of some greater points:

One must first set up Katana configuration structures with many of parameters that apply when Katana is used as CLI tool.
It is possible to do custom stuff in your callback function, such as registering the HTTP metadata and/or content somewhere for further processing.

Lesser known parts of Python standard library

Mon, 19 Aug 2024 00:00:00 +0000

In this article we will explore some lesser known, but interesting and useful corners of Python standard library.

Python dictionaries and lists are bread and butter for many applications, but might be too simple for more advanced data organisation. To provide more powerful containers for storing data in memory Python ships a collection module with things like:

Deque - list-like data structure for efficient insertion and removal of items from either end.
Counter - subclass of dictionary to count stuff.
ChainMap - meta-dictionary that provides unified key-value mapping across multiple dictionaries without copying the data.
Advanced structures for data mappings:
- OrderedDict - dictionary that maintains order of key-value pairs (e.g. when HTTP header value matters for dealing with certain security mechanisms).
- namedtuple - a factory function that creates a subclass of tuple with named fields. Could be convenient for more OOP approach than just using dict.
- defaultdict - dictionary with some or all or the values pre-filled. Could be handy when you want to provide values for fields that may be missing in web pages being scraped.
UserDict, UserList and UserString - wrappers for dict, list and str for further subclassing.

API reference: https://docs.python.org/3/library/collections.html

Python has with keyword to create context manager that will auto-cleanup things within lexical scope. For example, you don’t have to worry about closing a file yourself if you open it like this:

with open("results.txt", "w") as out_f:
    out_f.write("...\n")
    ...

contextlib module provide helpers to work with context managers in your code. For example, you can use @contextmanager annotation to turn your own function into one that could be used via with syntax.

API reference: https://docs.python.org/3/library/contextlib.html

Due to technical reality of how floating point number are represented, floating point computations can have some weird results:

>>> 0.1 + 0.1 + 0.1 - 0.3
5.551115123125783e-17
>>> 1.2 + 2.2
3.4000000000000004

This is mathematically incorrect and a deal breaker for some applications, such as those dealing with money. To address this problem, Python ships decimal module that allows us represent decimal numbers as Python objects with operator overloading. Computations are adjusted so that we get the correct result (within a configurable level of precision):

>>> from decimal import *
>>> one_point_two = Decimal('1.2')
>>> one_point_two
Decimal('1.2')
>>> two_point_two = Decimal('2.2')
>>> two_point_two
Decimal('2.2')
>>> one_point_two + two_point_two
Decimal('3.4')

API reference: https://docs.python.org/3/library/decimal.html

Likewise, we can run into problem when dealing with fractions - one third is not exactly equal to 0.333… Python ships with fractions module to represent them as Python objects with operator overloading:

>>> import fractions
>>> one_third = fractions.Fraction(numerator=1, denominator=3)
>>> 2 * one_third
Fraction(2, 3)

API reference: https://docs.python.org/3/library/fractions.html

Python interpreter is implemented as stack machine internally. When you run some Python code it is compiled to Python-specific bytecode that Python VM executes. For debugging or performance optimisation purposes it might be desirable to disassemble Python bytecode into instruction. The standard Python installation ships with dis module that takes Python code units (e.g. a function) and disassembles the code into a kind of pseudo-assembler:

>>> import dis
>>> def add(a, b):
...     return a+b
... 
>>> dis.dis(add)
  1           0 RESUME                   0

  2           2 LOAD_FAST                0 (a)
              4 LOAD_FAST                1 (b)
              6 BINARY_OP                0 (+)
             10 RETURN_VALUE

API reference: https://docs.python.org/3/library/dis.html

Python statistics module provides a small toolkit of statistical algorithms for relatively simple applications where it is an overkill to use Pandas or Numpy: standard deviation, several kinds of average of numeric data, linear regression, correlation, normal distribution and others. For people eager to join the AI/ML revolution the docs provide a sample code for making Naive Bayes classifier - an algorithm that can be considered a minimum viable example of machine learning.

API reference: https://docs.python.org/3/library/statistics.html

You may sometimes want to open web browser from your Python code. Since Python is portable and multiplatform language this may be bit of hassle due to differences between platforms and setups. To make this easier Python ships a webbrowser module with simple API to make a browser show a page. For example, we can call open_new() function with URL:

>>> import webbrowser
>>> webbrowser.open_new('https://trickster.dev')
True

If you want to open a local file (e.g. HTML report with results of some automation) you need to pass a file URL to one of the functions:

>>> import os
>>> file_url = 'file://' + os.path.realpath('test.html')
>>> webbrowser.open(file_url)
True

API reference: https://docs.python.org/3/library/webbrowser.html

Distributing Python programs in source code form can be problematic if there’s some non-technical users at the receiving end. Even if they have Python installed there can be lacking dependencies, version conflicts and other runtime issues. To make this less terrible Python ships zipapp module with CLI tool and Python API to package Python code into single file packages. This provides an interesting midpoint between using virtual environments and solutions like PyInstaller to compile the app into platform-specific form - code remains portable, but Python interpreter has to be there to run it.

API reference: https://docs.python.org/3/library/zipapp.html

Simple ways to find exposed sensitive information

Fri, 21 Jun 2024 00:00:00 +0000

Informational advantage is a form of power. On the flipside, exposure of sensitive information about a person or organisation can be a privacy/security problem. Sensitive Data Exposure is a type of vulnerability where software system (or user) makes sensitive data (API keys, user information, private documents, etc.) available to potential adversaries. For example, web app that lets users edit potentially confidential documents may be storing them in S3 buckets without any proper access controls. This results in information that should not be public being potentially available to those who know how to look for it. In this post we will go through some basic techniques of Sensitive Data Discovery - the activity of hunting for accidental leaks of things that are best kept hidden.

One way to look for potentially sensitive information is to do search engine dorking - launching queries specifically crafted to narrow down on specific, potentially sensitive things.

For example, the following query (Google dork) would look for PDF documents containing the word “confidential” and hosted under specific domain:

filetype:pdf site:hackerone.com "confidential"

We have used filetype: operator to limit result to specific file extension and site: operator to limit them to specific domain.

Of course, not every document containing word “confidential” is actually confidential, but a query like this could be a starting point to check if nothing sensitive is being leaked.

Another, more realistic example is the following Google query:

"contractors" filetype:xls "@gmail.com" site:gov

Automating Google dorking can be done via many of SERP scraper APIs available, but at the end of the day you will need to review the results manually.

On the very first page of search results I was able to find actual lists of government agency (e.g. Illinois Dept. of Transportation) contractors with email addresses. Since pretending to be a contractor is very viable social engineering vector that can lead to unauthorised access (physical and/or digital) to the sensitive infrastucture this sort of information exposure can have pretty serious security implications. Furthermore, service-based businesses should take care not to expose their lead list spreadsheet on the public web as they can potentially be found by some growth hacker working for the competition.

Real examples of sensitive, private information ending up findable via Google dorking:

H1 report 644358 - PDF files with SSNs and family data related to US military personnel.
H1 report 672629 - searching for site:*.mil ext:ppt intext:password would lead to US DoD training materials with usernames/password being available.

Another way to look for more technical sensitive data exposures is to search on Github.

For example, the following query (Github dork) looks for instances of Git credentials file being accidentally commited to public repos:

path:**/.git-credentials

We can also look for API keys to paid or private services. Github search query can be derived from sample code related to the API. Let me give you a real example. Google maps iOS SDK sample code repo contains the following code in MapsAndPlacesDemo/MapsAndPlacesDemo/ApiKeys.swift file:

/// API keys needed for this project
public enum ApiKeys {
    #error("Register for API keys and enter them below; then, delete this line")
    static let mapsAPI = ""
    static let placesAPI = ""
}

Now there is a chance that some people trying out the GMaps iOS SDK filled in the API keys and commited the code to public repository. Indeed, the following Github dork gives us what seem to be some real API keys to Places API:

"static let placesAPI"

However, it must be noted that Github has secret scanning automation running to warn users about accidentally publishing API keys for a growing list of partner services (and revoke leaked ones), which makes this trick less viable as time goes on.

But what if we want to find API keys and other sensitive pieces of information on websites running in production? In this case PublicWWW - a code-level search engine - can be helpful. To look for some API keys hardcoded into client-side JS snippets we can run queries like:

"api_key" depth:all
"apiKey" depth:all

Again, not everything you will find like this is going to be sensitive data. Some API keys are not meant to be kept secret.

Some real examples of API key exposures being a security problem:

H1 report 1065041 - Google geocoding API leak.
H1 report 1066410 - leaking API keys in client-side JS code lead to further security problems.
H1 report 1218754 - leaked API key lead to exposure of internal system statistics.

Like with Google, we can use site: operator to limit the results to given domain. However, PublicWWW results are limited to users without paid account and one must pay up to get the full results. Users with paid plan can also search for stuff via API.

Some sites heavily reliant on client side JavaScript may use preloaded state - a JSON document or JavaScript object with some initial data to be shown on the first load of webpage. It may include data beyond what is visibly seen on the screen. I have personally found and scraped email addresses from preloaded state that were not seen on the rendered form of HTML, but could be found by digging through the page source a little. In some cases, web pages have more information in them than what is seen at the very surface.

AWS S3 and compatible services (e.g. Digital Ocean Spaces, MinIO, and many others) are commonly used to store files in the cloud. For example, many web applications use it to store user-uploaded data that they are supposed to keep private - one user should not access other users files. However, sometimes due to misconfiguration S3 buckets happen to be wide open to anyone who finds them, which may lead to sensitive data discovery.

Some tools to look for misconfigured S3 buckets are:

Last, but not least way the sensitive data discovery happens I want to mention is people simply not keeping their mouths shut. See the following conference talks about this:

Remember: once you tell something to the world you no longer control that information. “Loose lips sink ships” says the ancient wisdom.

Understanding SOCKS protocol

Sat, 04 May 2024 00:00:00 +0000

In computer networking a proxy server is intermediate party (software that may or may not be running on a separate server) to transfer application-level traffic between client and another server - a real destination of the connection. There are two major ways this can be done: message forwarding (largely limited to plaintext HTTP) and connection tunneling. SOCKS is a widely utilised protocol that is based on the latter approach. Proxy servers may be useful for caching, load balancing, network traffic inspection, hiding real traffic source from target servers, bypassing censorship/geo-blocking and filtering the traffic. Today we will look how SOCKS protocol works to relay the traffic between client and target server.

There are two major versions of SOCKS protocol: legacy SOCKS4A and slightly newer, more advanced SOCKS5. Since SOCKS5 has some proper RFCs published to be used as specifications it will be our primary focus. At macro level, primary purpose of SOCKS5 protocol is to let user connect to a proxy server and provide a way to “outsource” a secondary connection originating from proxy server and terminated at the true destination the user wants to connect to. The original idea behind this was enabling network firewalls to provide a controlled access to servers running within private networks.

There are two document from IETF that we are most interested in:

RFC 1928 - SOCKS Protocol Version 5.
RFC 1929 - Username/Password Authentication for SOCKS V5.

Unlike many network protocol specifications, these are fairly short reads and can be consumed in a single sitting.

One key thing to notice here is that SOCKS5 is a binary protocol - unlike HTTP it does not rely on human readable strings, but uses shorter binary messages. The tradeoff here is that it uses bandwidth more efficiently at the expense of making troubleshooting more difficult.

When client establishes a connection to SOCKS5 server it first is supposed to send a version ID/method selection message with the following wire format:

VER (1 octet) - must be 0x05.
NMETHODS (1 octet) - how many auth methods will the client propose.
METHODS - a list of methods, represented by one octet each.

There are two auth methods that are of interest for us: no auth (0x00) and username/password auth (0x02). There’s also GSSAPI: a legacy API that is technically required by the spec to allow for secure cryptographic handshake, but let’s not go there now.

So the following are valid examples of this message:

05 01 00 - client proposes doing connection without auth.
05 01 02 - client proposes using username/password based auth.
05 02 00 02 - client proposes either of the two and letting the server choose.

Notice the TLV structure being used here: first we have “tag” - protocol version number; next we have length - how many methods will be provided next and last we have a list of methods. Structuring the wire format this way enables parsing to be done incrementally.

Now the server has to make a choice if it wants to proceed with the connection and respond with method selection consisting of two fields, one octet each:

VER - must be 0x05 again.
METHOD - either one of the proposed method values or 0xFF if all are rejected (connection is closed in that case).

If no-auth (0x00) is chosen the client is now allowed to proceed with using the proxying features of the server. But if username/password based auth is negotiated there is one extra thing left to do - the credential-based subnegotiation. Again, there are two messages being exchanged between client and server. Client sends the username/password request of the following wire format:

VER - must be 0x01.
ULEN - username length (without trailing NUL character).
UNAME - username as ASCII string (ULEN octets).
PLEN - password length.
PASSWD - password.

Note that credentials are transferred in plaintext. Thus care must be taken when considering using SOCKS in potentially hostile networks. Don’t end up on Wall of Sheep during DEFCON!

Upon receipt of this message, server checks the credentials and responds with this kind of response (fields are 1 byte each):

VER - must be 0x01 again.
STATUS - 0 iff auth succeeded, some other value otherwise.

Server closes the connection if client does not provide credentials that check out.

Now we are ready to the real deal - establishing a connection to the remote via proxy server. The client sends a request of the following wire format:

VER (1 octet) - protocol version (5).
CMD (1 octet) - command to the server (0x01 if we want to proxy a TCP connection).
RSV (1 octet) - reserved, must be 0x00.
ATYP - address type:
- 0x01 - IPv4
- 0x03 - DNS hostname.
- 0x04 - IPv6
DST.ADDR - destination address (in NBO); size depends on ATYP.
DST.PORT (2 octets) - destination port (in NBO).

The server will attempt to service the request. In typical use case CMD will be 0x01 and that will be a CONNECT request asking the proxy to establish TCP connection to remote host and bridge (splice) the TCP streams.

When SOCKS server succeeds (or fails) to handle the request it will give one last response:

VER - 0x05.
REP - reply field; 0x00 on success, error code on failure.
RSV - reserved, must be 0x00.
ATYP - address type.
BND.ADDR - address the server is bound to (e.g. exit IP of outbound connection).
BND.PORT - port of outbound connection.

If CONNECT request succeeded the client can now use the resulting two-legged connection as if it was a regular TCP connection - for all the remote server knows it just got a fresh connection from a client and is ready to talk TLS or some other protocol that is based on TCP transport. Furthermore, if remote server is another SOCKS server it is possible to do a new SOCKS handshake for the purpose of proxy chaining - extending the connection via one more proxy.

Besides the outbound TCP connection support, there are couple more slightly esoteric features SOCKS protocol has. One is BIND command (0x02) that lets the client use SOCKS server for a secondary incoming connection. In that case BND.ADDR and BND.PORT fields in the reply tell the client where in the network the port has been opened.

It is meant to be used with legacy FTP servers that work in active mode by establishing the second connection to transfer contents (files, directory listings) while leaving the original connection user originated available for new commands. Since the RFC requires the initial primary connection to be present for the lifetime of secondary connection this feature is not very useful for anything else.

Second interesting, but not very used SOCKS feature is UDP support. If a client sends UDP ASSOCIATE command (CMD = 0x03) the server may bind UDP port and setup relaying of UDP datagrams. Since there is no such thing as connection at UDP level and because it is desirable to retain the original port numbers in UDP datagrams they are encapsulated in a wrapper message (defined in Section 7 of RFC 1928) when transferred between proxy and the client.

Some interesting software projects supporting (subset of) SOCKS protocol:

Tor - overlay network for anonymous communications.
OpenSSH - SOCKS can be used over SSH tunnel. See my previous post on SSH tips and tricks.
Curl - see --socks5, --proxy, --socks5-basic CLI options.
Python requests module if pysocks is installed.
Mitmproxy can be used in SOCKS5 mode (e.g. mitmdump --mode socks5).
Proxychains-ng is Unix/Linux tool to tunnel connections through one or more proxies.

Scraping Clutch for B2B company data

Fri, 12 Apr 2024 00:00:00 +0000

Clutch.co is a web portal serving as B2B service company directory. One may want to scrape it to make a list of companies within a service niche to target them with some form of outreach. Company name, website URL, phone number, hourly rate, project size, social media URLs would be fields of the dataset we would create by scraping this site. But the thing is, Clutch is fighting scraping attempts by using Cloudflare antibot service - naively fetching a page from this site gives us a 403 response with JS challenge. So what do we do?

Some would suggest using anti-detect browser as foundation for the scraper, but let us not rush to that. Instead, let’s work out what makes or breaks a request here. Open the site in Chrome, accept the cookies and let it pass the JS challenge. Now, by using “Copy as cURL” feature in Chrome DevTools we can get something like the following curl(1) command:

curl 'https://clutch.co/profile/testmatick' \
  -H 'authority: clutch.co' \
  -H 'accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7' \
  -H 'accept-language: en-GB,en-US;q=0.9,en;q=0.8' \
  -H 'cache-control: no-cache' \
  -H 'cookie: CookieConsent={stamp:%27gy8cYhc5rqyD+4wboFTjYwRZ3kmocvJU5dWpISe25EAYM1scKDNvWg==%27%2Cnecessary:true%2Cpreferences:false%2Cstatistics:false%2Cmarketing:false%2Cmethod:%27explicit%27%2Cver:1%2Cutc:1695021963851%2Cregion:%27lt%27}; exp_new-ab-test_attribute=1; exp_primary-cta-ab-test_attribute=old-main-CTA; __cf_bm=1Fxjs7RHjDX7mJRgzYLLodoV34MLcm0wFywRiYuqI0A-1708179657-1.0-ARy825yNViMh5Y/1qk+ul7/3KcNwb3kyyLNo+euY/dSC3f0123QWFZicuEcWZHDvEI0PFFzGb2A3IH++XuIto7c=; cf_clearance=2vwV27CEWNrT84e_nLo96kgOoihjSA2iBGBHBJvehAs-1708179658-1.0-AUS2KK3KRx6wX63hAVmC5YVDwczAz6d90pitbdBM6BIixlgYM/x1QaBMkzbT34dS2Trav72zccQ/zlvmoyicerE=' \
  -H 'pragma: no-cache' \
  -H 'sec-ch-ua: "Not A(Brand";v="99", "Google Chrome";v="121", "Chromium";v="121"' \
  -H 'sec-ch-ua-mobile: ?0' \
  -H 'sec-ch-ua-platform: "macOS"' \
  -H 'sec-fetch-dest: document' \
  -H 'sec-fetch-mode: navigate' \
  -H 'sec-fetch-site: none' \
  -H 'sec-fetch-user: ?1' \
  -H 'upgrade-insecure-requests: 1' \
  -H 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36' \
  --compressed

Running this command in the shell does indeed give us a proper page, so it seems neither TLS nor HTTP/2 fingerprinting is used. We see some browser-specific headers here and we see some cookies being used (at least not to block cURL). To narrow down what exactly is needed, we can remove some headers and relaunch the request - we discover that only cookies and user-agent header is necessary. Now we can remove cookies one by one and repeat the same kind of experiment. We find that both of exp_ cookies can be removed without triggering the antibot.

So the cookies that are necessary seem to be:

CookieConsent
__cf_bm
cf_clearance

The latter two are directly related to Cloudflare’s antibot products and are of prime importance to our objective.

We need to get these cookies somehow. Let us try using Bright Data’s Scraping Browser - a programmatically controllable web browser hosted for us in blocking-resistant setup. Despite being hosted in the cloud it can be integrated with the usual suspects of browser automation - Selenium, Playwright, Puppeteer. But we don’t need to implement the entire scraping flow with browser automation - we merely have to use it just enough to get the CF cookies.

Assuming we have Scraping Browser configured on Bright Data customer UI let us rework the sample code from documentation page to suit our objective:

#!/usr/bin/python3

import asyncio
from playwright.async_api import async_playwright

# $ pip3 install playwright

from pprint import pprint

USERNAME = "[REDACTED]"
PASSWORD = "[REDACTED]"

AUTH = USERNAME + ":" + PASSWORD
SBR_WS_CDP = f"wss://{AUTH}@brd.superproxy.io:9222"


async def run(pw):
    print("Connecting to Scraping Browser...")
    browser = await pw.chromium.connect_over_cdp(SBR_WS_CDP)
    try:
        print("Connected! Navigating...")
        page = await browser.new_page()
        await page.goto("https://clutch.co/profile/devfortress", timeout=2 * 60 * 1000)
        cookies = await page.context.cookies()
        pprint(cookies)
    except Exception as e:
        print(e)
    finally:
        await browser.close()


async def main():
    async with async_playwright() as playwright:
        await run(playwright)


if __name__ == "__main__":
    asyncio.run(main())

We can easily convert this snippet to use synchronous Playwright API:

#!/usr/bin/python3

from playwright.sync_api import sync_playwright, Playwright

from pprint import pprint

USERNAME = "[REDACTED]"
PASSWORD = "[REDACTED]"

AUTH = USERNAME + ":" + PASSWORD
SBR_WS_CDP = f"wss://{AUTH}@brd.superproxy.io:9222"

def run(pw: Playwright):
    browser = pw.chromium.connect_over_cdp(SBR_WS_CDP)
    try:
        page = browser.new_page()
        page.goto("https://clutch.co", timeout=1 * 60 * 1000)
        cookies = page.context.cookies()
        pprint(cookies)
    except Exception as e:
        print(e)
    finally:
        browser.close()

def main():
    with sync_playwright() as playwright:
        run(playwright)

if __name__ == "__main__":
    main()

Once we got the cookies, we can proceed with scraping the data. There are three levels to the scraping process we are going to implement:

Grabbing a list of business categories.
Traversing this list will give us bunch of links for company pages (each category URL points to a paged list of companies within a category - we have to account for some overlap between categories).
Scraping each company page will give us an item - a unit of data that is to be saved into CSV file.

Since this involves cookies that we have to reuse between requests we want to set up a requests session to use for scraping and recreate it when needed. However there is a little complication - it turned out that TLS fingerprint is actually being checked, thus we need to use TLS client library to reproduce a fingerprint that looks realistic - like the one coming from real browser, not from Python requests library (we got lucky with curl earlier). We thus use Playwright together with open source TLS client (tls-client from PIP) to prepare an equivalent of requests.Session object:

import tls_client
from playwright.sync_api import sync_playwright, Playwright

USERNAME = "[REDACTED]"
PASSWORD = "[REDACTED]"
PROXY_URL = "[REDACTED]"

AUTH = USERNAME + ":" + PASSWORD
SBR_WS_CDP = f"wss://{AUTH}@brd.superproxy.io:9222"


def get_cookies(page_url):
    cookies = None

    with sync_playwright() as pw:
        browser = pw.chromium.connect_over_cdp(SBR_WS_CDP)
        try:
            page = browser.new_page()
            page.goto(page_url, timeout=1 * 60 * 1000)
            cookies = page.context.cookies()
            print("Got cookies:", cookies)
        except Exception as e:
            print(e)
        finally:
            browser.close()

    return cookies


def create_session(page_url):
    session = tls_client.Session(
        client_identifier="chrome120", random_tls_extension_order=True
    )

    session.headers = {
        "accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7",
        "accept-language": "en-GB,en-US;q=0.9,en;q=0.8",
        "cache-control": "no-cache",
        "pragma": "no-cache",
        "sec-ch-ua": '"Google Chrome";v="123", "Not:A-Brand";v="8", "Chromium";v="123"',
        "sec-ch-ua-mobile": "?0",
        "sec-ch-ua-platform": '"macOS"',
        "sec-fetch-dest": "document",
        "sec-fetch-mode": "navigate",
        "sec-fetch-site": "none",
        "sec-fetch-user": "?1",
        "upgrade-insecure-requests": "1",
        "user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 14_2_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.6099.234 Safari/537.36",
    }

    cookies = get_cookies(page_url)

    for cookie in cookies:
        session.cookies.set(
            cookie.get("name"), cookie.get("value"), domain=cookie.get("domain")
        )

    session.proxies = {"http": PROXY_URL, "https": PROXY_URL}

    return session

When creating session we also set browser-like HTTP headers and integrate a proxy service - in this case DC proxies are sufficient as long as we have the proper cookies. Nobody seems to care that CF challenge was solved from different IP than the requests we generate later (up to a point - sometimes we need to redo the session creation step to proceed with scraping).

I elected to break down the scraping process into two scripts. The first one sources list of companies by first scraping Sitemap page, then traversing the company lists to make a deduplicated list of profile URLs. Not much to see there if you have read more of the posts on this blog. The second script goes through profile URLs and scrapes some information from profile pages:

#!/usr/bin/python3

import csv
from concurrent.futures import ThreadPoolExecutor
from datetime import datetime
from pprint import pprint
from urllib.parse import unquote
import time
import threading

from lxml import html

from tls_session import create_session, safe_get

sessions = dict()

def prepare_thread_session():
    sessions[threading.current_thread()] = create_session("https://clutch.co")

def scrape_company_page(company_url):
    session = sessions[threading.current_thread()]
    resp = safe_get(session, company_url)
    if resp is None or resp.status_code == 403:
        print("Retrying: {}".format(company_url))
        prepare_thread_session()
        session = sessions[threading.current_thread()]
        resp = safe_get(session, company_url)

    if resp is None:
        return None

    print(resp.url, resp.status_code)

    if resp.status_code != 200:
        return None

    tree = html.fromstring(resp.text)

    company_name = tree.xpath("//h1/a/text()")
    if len(company_name) == 1:
        company_name = company_name[0].strip()
    else:
        company_name = None

    # ...

    row = {
        "company_name": company_name,
        # ...
    }

    return row

FIELDNAMES = [
    "company_name",
    # ...
]


def main():
    company_urls = []

    in_f = open("lists.csv", "r")
    
    csv_reader = csv.DictReader(in_f)

    for row in csv_reader:
        company_urls.append(row['company_url'])

    in_f.close()
    
    out_f = open("pages.csv", "w+", encoding="utf-8")
    
    csv_writer = csv.DictWriter(out_f, fieldnames=FIELDNAMES, lineterminator='\n')
    csv_writer.writeheader()

    with ThreadPoolExecutor(16, initializer=prepare_thread_session) as executor:
        for row in executor.map(scrape_company_page, company_urls):
            if row is None:
                continue
            pprint(row)
            csv_writer.writerow(row)

    out_f.close()

if __name__ == "__main__":
    main()

Some things to note here:

ThreadPoolExecutor object from concurrent.futures was used to create and run a thread pool to make the scraping concurrent.
For each thread we need a session that we initially create in thread initializer function and also recreate on as-needed basis. The sessions dictionary is keyed by the thread which allows each thread to unambiguously access the session it created.
Sometimes HTTPS requests fail with some exception (e.g. connection times out). To prevent this from crashing the entire script we use safe_get() helper function.

By using rather simple code I was able to scrape over 307 000 rows of company data with pretty trivial opex and not much development effort. The scraping can be done overnight in a small VPS despite the first script being single-threaded and the second one using only 16 threads.

Scraped data can be further enriched via automated Google dorking or enrichment APIs to build contact lists for lead generation. One can also do some some additional scraping to extract company emails from their websites.

SSH tips and tricks

Sun, 04 Feb 2024 00:00:00 +0000

SSH is well established protocol for securely accessing remote systems over the network for administration and devops purposes. A widely deployed OpenSSH software suite implements this protocol. But there is more to the SSH technology than reaching a remote shell over ssh(1) or copying files via scp(1)/sftp(1). We will go through some lesser known, somewhat advanced tricks and use cases that could be valuable in your daily work.

SSH escape sequences

Most people don’t know that ssh(1) provides somewhat secret control sequences that can be used to manage in-progress connections. Make sure the last keystroke you have typed is Enter, then type the tilde character and question mark (~?). You will be given the following message listing other sequences you can do in a similar way:

Supported escape sequences:
 ~.   - terminate connection (and any multiplexed sessions)
 ~B   - send a BREAK to the remote system
 ~C   - open a command line
 ~R   - request rekey
 ~V/v - decrease/increase verbosity (LogLevel)
 ~^Z  - suspend ssh
 ~#   - list forwarded connections
 ~&   - background ssh (when waiting for connections to terminate)
 ~?   - this message
 ~~   - send the escape character by typing it twice
(Note that escapes are only recognized immediately after newline.)

~C gives you a secondary command prompt that can be used to manage port forwarding/tunneling (more on this later).

Single command through SSH

Sometimes you want to run just a single OS shell command via SSH connection and get the output. That can be done with a standard SSH client by launching it with -t and passing your command as last argument:

$ ssh -t root@167.172.24.25 "date -R"    
Sun, 14 Jan 2024 16:52:37 +0000
Connection to 167.172.24.25 closed.

Generally speaking, it is a recommended practice to use SSH keys for authentication and to avoid relying on passwords. Passwordless SSH effectively prevents any successful brute force attack that could be done over the network. However, if for some reason you still want to use SSH interface with password you can make it more secure by adding Two Factor Authentication step. We will go through the steps that can be taken to set it up on Debian 12 system.

Make sure your APT cache is up to date and install libpam-google-authenticator package through APT:

# apt-get install libpam-google-authenticator

Now we need to edit /etc/pam.d/sshd file to add the following line there:

auth required pam_google_authenticator.so

Also edit /etc/ssh/sshd_config to make sure there’s following lines there (watch out for conflicting directives - you may need to change no to yes on some lines):

UsePAM yes
ChallengeResponseAuthentication yes
KbdInteractiveAuthentication yes
AuthenticationMethods keyboard-interactive

Restart the sshd daemon to force it to read the updated config file:

# systemctl restart sshd.service

In your mobile device install Google Authenticator or a compatible app (e.g. FreeOTP.

Now we’re ready to run the google-authenticator tool, answer some questions and follow the instructions:

# google-authenticator 

Do you want authentication tokens to be time-based (y/n) y
Warning: pasting the following URL into your browser exposes the OTP secret to Google:
  https://www.google.com/chart?chs=200x200&chld=M|0&cht=qr&chl=otpauth://totp/root@debian-s-1vcpu-512mb-10gb-ams3-01%3Fsecret%3D[REDACTED]%26issuer%3Ddebian-s-1vcpu-512mb-10gb-ams3-01
                                                                                                          
[REDACTED]

Your new secret key is: [REDACTED]
Enter code from app (-1 to skip): 371342
Code confirmed
Your emergency scratch codes are:
  [REDACTED]

Do you want me to update your "/root/.google_authenticator" file? (y/n) y

Do you want to disallow multiple uses of the same authentication
token? This restricts you to one login about every 30s, but it increases
your chances to notice or even prevent man-in-the-middle attacks (y/n) y

By default, a new token is generated every 30 seconds by the mobile app.
In order to compensate for possible time-skew between the client and the server,
we allow an extra token before and after the current time. This allows for a
time skew of up to 30 seconds between authentication server and client. If you
experience problems with poor time synchronization, you can increase the window
from its default size of 3 permitted codes (one previous code, the current
code, the next code) to 17 permitted codes (the 8 previous codes, the current
code, and the 8 next codes). This will permit for a time skew of up to 4 minutes
between client and server.
Do you want to do so? (y/n) y

If the computer that you are logging into isn't hardened against brute-force
login attempts, you can enable rate-limiting for the authentication module.
By default, this limits attackers to no more than 3 login attempts every 30s.
Do you want to enable rate-limiting? (y/n) y

We were given the option to either scan the QR code rendered on the terminal in colored blocks or to have it displayed in the browser.

Screenshot 1

Once we do all of this, we will be asked both one time key and the password when logging in via SSH:

$ ssh root@178.62.232.185
(root@178.62.232.185) Verification code: 
(root@178.62.232.185) Password:

SSH tunneling with `-L`

Sometimes you have some networked software running on remote system with listener socket being bound to localhost or otherwise restricted network interface.

One example is the situation in Jupyter notebook VPS on Digital Ocean. Once you instantiate the VPS image you can run notebook.sh script in the home directory of ubuntu user to start the Jupyter lab webapp which will bind to localhost for security purposes. However, we are instructed to use ssh -L to access it through SSH tunnel:

$ bash notebook.sh start
JupyterLab started.
See the logs under ~/notebook.log.

Use 'notebook list' to view the URL.

Remember that you need to create a tunnel to access the notebook from your localhost.
On another terminal, use ssh root@<ip> -L 8888:localhost:8888 to create the tunnel. Then you can access the notebook from your local browser.

Running the provided command on client-side shell gives another SSH session with the added benefit of Jupyter web app being available locally at port 8888.

Screenshot 2

As another example, a Linux server running LAMP stack may allow HTTP and SSH access on public network, but make MySQL listen for incoming connections on 127.0.0.1. That poses an inconvenience if we want to connect to the MySQL server over the network. In cases like this, SSH tunneling - i.e. wrapping non-SSH traffic in SSH connection can be quite useful.

SSH connection as local SOCKS proxy

SSH tunneling enables us to access a single network daemon that otherwise would not be reached. But what if we want to reach everything that can be reached from the server? In that case we can set up SSH connection as local SOCKS proxy with command like:

$ ssh -D 1080 root@64.23.163.194

Here 1080 is the TCP port on local network interface we want to open as SOCKS proxy.

By using curl, we can verify that it works as intended:

$ curl -s -x socks5://localhost:1080 https://lumtest.com/myip.json | jq
{
  "ip": "64.23.163.194",
  "country": "US",
  "asn": {
    "asnum": 14061,
    "org_name": "DIGITALOCEAN-ASN"
  },
  "geo": {
    "city": "Santa Clara",
    "region": "CA",
    "region_name": "California",
    "postal_code": "95054",
    "latitude": 37.3931,
    "longitude": -121.962,
    "tz": "America/Los_Angeles",
    "lum_city": "santaclara",
    "lum_region": "ca"
  }
}

This can also done via the “secret” SSH command line (~C) during an in-progress SSH session:

ssh> -D 1080
Forwarding port.

SSH server behind NAT via Tor Onion Service

Tor is a well-known distributed system for anonymous communications that allows for a convenient secondary use case: NAT traversal via Onion Services. Typically this is done to expose a website or web app to be reachable via Tor Browser, but since Tor works at TCP connection level nothing is stopping us from setting up Onion Service for some TCP-based application other than HTTP - i.e. we can also use it with SSH.

For example, on Debian 12 server we could do as follows.

We start by installing Tor:

# apt-get update
# apt-get install -y tor

Let us create a directory somewhere in file system to store the cryptographic keys and metadata:

# mkdir /var/onionservice

Limit the access to this directory so that only owner can fully access it:

# chmod 700 /var/onionservice/

Edit your torrc file (on Debian it’s available at /etc/tor/torrc) to add the following directives:

HiddenServiceDir /var/onionservice
HiddenServicePort 22 127.0.0.1:22

Restart or launch the tor daemon (e.g systemctl restart tor). If all is good the Onion Service directory will have some files, including the one named hostname:

# cat /var/onionservice/hostname 
iyqklkzlvieyehybr44yqyfqavcvhepdltumngxfje7x36ewx2li3hyd.onion

Now we need to tunnel the SSH connection over Tor at client side. This can be done via program named torsocks:

$ torsocks ssh root@iyqklkzlvieyehybr44yqyfqavcvhepdltumngxfje7x36ewx2li3hyd.onion

The caveat of this approach it that the extra latency Tor introduces can make the shell work very laggy and slow. This can be particulary bad for touch typers who rely on visual feedback to fix typos as they happen. A possible improvement is making your Onion Service use only one hop into Tor network by adding the following directive into torrc:

HiddenServiceNonAnonymousMode 1
HiddenServiceSingleHopMode 1

However it must be noted that this essentially removes the server hiding aspect (which is fine if NAT traversal is all you need). This is also incompatible with SOCKS interface being open on Tor instance at your server.

Torsocks is available via package managers on some Linux/Unix systems (e.g. Debian Linux, macOS).

Mounting directory subtree via sshfs

SSHFS is a program that mounts a remote file system subtree to local one over SFTP connection. It is part of FUSE project that ships a framework for developing custom file systems.

The following is assuming you are using Debian 12 on client side of the connection.

Install sshfs via APT:

# apt-get update
# apt-get install -y sshfs

On client side create the new directory to be used via sshfs:

$ mkdir /mnt/droplet

To mount a remote FS subtree via sshfs now we can run a command like:

$ sshfs root@209.97.131.201:/etc /mnt/droplet

When we no longer need access to remove directory we can use umount(1) to unmount the directory we have created:

$ umount /mnt/droplet

SSHFS is mostly supported on Linux. For other platforms there are some alternative solutions:

macFUSE project ships it’s own version of sshfs for macOS.
sshfs-win is somewhat limited port of sshfs for Windows that relies on WinFsp - an equivalent of FUSE for Windows.

Your mileage may vary.

Using SSH programmatically via Paramiko/Fabric

Paramiko is a Python module that implements SSH protocol. It provides a somewhat low level API for automating SSH tasks. For example, connecting to remote SSH server and running a command can be done in few lines of code:

#!/usr/bin/python3

import sys

import paramiko


def main():
    if len(sys.argv) != 5:
        print("{} <hostname> <username> <password> <command>".format(sys.argv[0]))
        return

    hostname = sys.argv[1]
    username = sys.argv[2]
    password = sys.argv[3]
    command = sys.argv[4]

    client = paramiko.client.SSHClient()

    client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
    client.connect(hostname, username=username, password=password)

    _, stdout, _ = client.exec_command(command)

    output = stdout.read().decode("utf-8")

    print(output)


if __name__ == "__main__":
    main()

Fabric is Python module that wraps Paramiko to provide a more abstract, higher level API.

Retrospective scraping of historical data from Internet Archive snapshots

Wed, 27 Dec 2023 00:00:00 +0000

Some internet research activities are based not only on the present data, but also on historical data that was, but no longer is posted online. As of late 2023, Internet Archive has over 842 billion web page snapshots stored and available for retrieval. We will go through a simple example of how scraping pre-crawled pages from Wayback Machine can be used to gather historical data for data science purposes. To look into Iowa inmate population trends, we will be scraping historical snapshots of Iowa Department of Corrections inmate statistics page. Data of interest is in “Current Count” and “Institution” columns of the table. We will do data scraping and wrangling in Python with Pandas dataframe library.

To quickly view the available snapshots for this page we search it through Archive.org front page. But there seems to be a little complication. Only two snapshots are available, which would not be sufficient for our purposes.

Screenshot 1

Thus we must backtrack a little. Searching for the domain - doc-search.iowa.gov - on Wayback Machine gives us only 5 snapshots. It seems this domain is pointing to fairly new (sub)site that is primarily meant for looking up prisoners in Iowa and only secondarily to provide overall inmate statistics. The primary domain for Iowa Dept. of Corrections is doc.iowa.gov. Exploring snapshots for this site reveals that until some time in 2023 it had a statistics page at following URL:

https://doc.iowa.gov/daily-statistics

Screenshot 2 Screenshot 3

We are after data on all available snapshots of this page.

To check for snapshot availability we can call the /wayback/available API endpoint with the URL of interest:

$ curl "https://archive.org/wayback/available?url=https://doc.iowa.gov/daily-statistics" | jq
{
  "url": "https://doc.iowa.gov/daily-statistics",
  "archived_snapshots": {
    "closest": {
      "status": "200",
      "available": true,
      "url": "http://web.archive.org/web/20230429064712/https://doc.iowa.gov/daily-statistics",
      "timestamp": "20230429064712"
    }
  }
}

How do we get the list of snapshots? Internet Archive CDX API is a flexible way to search for snapshots by URL and secondary criteria: HTTP status code, timeframe and MIME type of the content. By default this API provides output in CDX table format that looks as follows:

$ curl "http://web.archive.org/cdx/search/cdx?url=doc.iowa.gov&limit=5"
gov,iowa,doc)/ 20170323172420 https://doc.iowa.gov/ text/html 200 L4NPH6MSVTZBRCD6CUTTQ2Y66OKY5Y3S 6904
gov,iowa,doc)/ 20170323172435 https://doc.iowa.gov/ text/html 200 BPFQVM2VIW32BNVM7LJ23QTXN7TDTPIL 6906
gov,iowa,doc)/ 20170323172720 https://doc.iowa.gov/ text/html 200 76LUVXUPNL3NPGU246YM3YQ6VNOOVIVH 6907
gov,iowa,doc)/ 20170416221430 https://doc.iowa.gov/ text/html 200 6VMMWJRPN33DAGREX54WSLFG367NCCEF 7113
gov,iowa,doc)/ 20170417071150 https://doc.iowa.gov/ text/html 200 OZMIM56F6OM5IL3HXTWBJZQCMGVRBYTB 7119

We can get a JSONified version of this table by passing output=json URL parameter:

$ curl "http://web.archive.org/cdx/search/cdx?url=doc.iowa.gov&limit=5&output=json"
[["urlkey","timestamp","original","mimetype","statuscode","digest","length"],
["gov,iowa,doc)/", "20170323172420", "https://doc.iowa.gov/", "text/html", "200", "L4NPH6MSVTZBRCD6CUTTQ2Y66OKY5Y3S", "6904"],
["gov,iowa,doc)/", "20170323172435", "https://doc.iowa.gov/", "text/html", "200", "BPFQVM2VIW32BNVM7LJ23QTXN7TDTPIL", "6906"],
["gov,iowa,doc)/", "20170323172720", "https://doc.iowa.gov/", "text/html", "200", "76LUVXUPNL3NPGU246YM3YQ6VNOOVIVH", "6907"],
["gov,iowa,doc)/", "20170416221430", "https://doc.iowa.gov/", "text/html", "200", "6VMMWJRPN33DAGREX54WSLFG367NCCEF", "7113"],
["gov,iowa,doc)/", "20170417071150", "https://doc.iowa.gov/", "text/html", "200", "OZMIM56F6OM5IL3HXTWBJZQCMGVRBYTB", "7119"]]

This is still not quite the shape of data that most modern JSON APIs provide, but we can work with that by using read_json() function from Pandas:

$ python3
Python 3.11.6 (main, Oct  2 2023, 20:46:14) [Clang 14.0.3 (clang-1403.0.22.14.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas as pd
>>> df = pd.read_json("http://web.archive.org/cdx/search/cdx?url=doc.iowa.gov&limit=5&output=json")
>>> df
                0               1                      2          3           4                                 5       6
0          urlkey       timestamp               original   mimetype  statuscode                            digest  length
1  gov,iowa,doc)/  20170323172420  https://doc.iowa.gov/  text/html         200  L4NPH6MSVTZBRCD6CUTTQ2Y66OKY5Y3S    6904
2  gov,iowa,doc)/  20170323172435  https://doc.iowa.gov/  text/html         200  BPFQVM2VIW32BNVM7LJ23QTXN7TDTPIL    6906
3  gov,iowa,doc)/  20170323172720  https://doc.iowa.gov/  text/html         200  76LUVXUPNL3NPGU246YM3YQ6VNOOVIVH    6907
4  gov,iowa,doc)/  20170416221430  https://doc.iowa.gov/  text/html         200  6VMMWJRPN33DAGREX54WSLFG367NCCEF    7113
5  gov,iowa,doc)/  20170417071150  https://doc.iowa.gov/  text/html         200  OZMIM56F6OM5IL3HXTWBJZQCMGVRBYTB    7119

It misread column labels in the first line, but we can trivially fix it:

>>> headers = df.iloc[0]
>>> df = pd.DataFrame(df.values[1:], columns=headers)
>>> df
0          urlkey       timestamp               original   mimetype statuscode                            digest length
0  gov,iowa,doc)/  20170323172420  https://doc.iowa.gov/  text/html        200  L4NPH6MSVTZBRCD6CUTTQ2Y66OKY5Y3S   6904
1  gov,iowa,doc)/  20170323172435  https://doc.iowa.gov/  text/html        200  BPFQVM2VIW32BNVM7LJ23QTXN7TDTPIL   6906
2  gov,iowa,doc)/  20170323172720  https://doc.iowa.gov/  text/html        200  76LUVXUPNL3NPGU246YM3YQ6VNOOVIVH   6907
3  gov,iowa,doc)/  20170416221430  https://doc.iowa.gov/  text/html        200  6VMMWJRPN33DAGREX54WSLFG367NCCEF   7113
4  gov,iowa,doc)/  20170417071150  https://doc.iowa.gov/  text/html        200  OZMIM56F6OM5IL3HXTWBJZQCMGVRBYTB   7119

From the timestamp column we can derive the snapshot URL by treating the timestamp as path component between https://web.archive.org/web/ and the original URL:

>>> df['snapshot_url'] = df['timestamp'].apply(lambda ts: 'https://web.archive.org/web/' + ts + '/https://doc.iowa.gov/')
>>> df['snapshot_url'].to_list()
['https://web.archive.org/web/20170323172420/https://doc.iowa.gov/', 'https://web.archive.org/web/20170323172435/https://doc.iowa.gov/', 'https://web.archive.org/web/20170323172720/https://doc.iowa.gov/', 'https://web.archive.org/web/20170416221430/https://doc.iowa.gov/', 'https://web.archive.org/web/20170417071150/https://doc.iowa.gov/']

Each snapshot is the original HTML page with some extra stuff injected to help us navigate the history of the page. Thus we can scrape it by using all the usual web scraping techniques. Furthermore, the old pages we want to scrape present the data in the basic HTML <table> element, which makes it easy to scrape by using Pandas read_html() function:

>>> dfs = pd.read_html("https://web.archive.org/web/20210303210751/https://doc.iowa.gov/daily-statistics")
>>> stats_df = dfs[1]
>>> stats_df
                      Institution Current Count  Capacity  Medical/Segregation
0                         Anamosa           943     911.0                171.0
1                        Clarinda           962     750.0                 46.0
2                      Fort Dodge          1151    1162.0                 75.0
3                   Mitchellville           486     654.0                114.0
4                Minimum Live-Out            99     120.0                  0.0
5                         Oakdale           818     585.0                142.0
6   Forensic Psychiatric Hospital             9       0.0                 28.0
7                    Fort Madison           708     612.0                153.0
8                  Mount Pleasant           815     776.0                 80.0
9                   Newton-Medium           853     762.0                197.0
10                        Minimum           175     252.0                  6.0
11                  Rockwell City           432     245.0                 18.0
12          MPCF Minimum Live-Out            86     104.0                  0.0
13           INSTITUTIONAL TOTALS          7537    6933.0               1030.0
14               % overcrowded by         8.71%       NaN                  NaN

To clean up the data, let us drop the last two columns and last two rows:

>>> stats_df = stats_df[['Institution', 'Current Count']]
>>> stats_df = stats_df[:-2]
>>> stats_df
                      Institution Current Count
0                         Anamosa           943
1                        Clarinda           962
2                      Fort Dodge          1151
3                   Mitchellville           486
4                Minimum Live-Out            99
5                         Oakdale           818
6   Forensic Psychiatric Hospital             9
7                    Fort Madison           708
8                  Mount Pleasant           815
9                   Newton-Medium           853
10                        Minimum           175
11                  Rockwell City           432
12          MPCF Minimum Live-Out            86

Lastly, let us tweak names of some sub-entries:

>>> stats_df.loc[stats_df['Institution'] == "Forensic Psychiatric Hospital", 'Institution'] = "Oakdale - Forensic Psychiatric Hospital"
>>> stats_df.loc[stats_df['Institution'] == "Minimum", 'Institution'] = "Newton-Minimum"
>>> stats_df.loc[stats_df['Institution'] == "Minimum Live-Out", 'Institution'] = "Mitchellville - Minimum Live-Out"
>>> stats_df
                                Institution Current Count
0                                   Anamosa           943
1                                  Clarinda           962
2                                Fort Dodge          1151
3                             Mitchellville           486
4          Mitchellville - Minimum Live-Out            99
5                                   Oakdale           818
6   Oakdale - Forensic Psychiatric Hospital             9
7                              Fort Madison           708
8                            Mount Pleasant           815
9                             Newton-Medium           853
10                           Newton-Minimum           175
11                            Rockwell City           432
12                    MPCF Minimum Live-Out            86

By now we have prototyped finding and parsing page snapshots in Python REPL environment. Let us put everything together into a Python script:

#!/usr/bin/python3

from urllib.parse import urlencode
import time
import sys

import pandas as pd

ORIG_URL = "https://doc.iowa.gov/daily-statistics"


def get_snapshot_urls():
    params = {"url": ORIG_URL, "output": "json", "filter": "statuscode:200"}

    cdx_url = "http://web.archive.org/cdx/search/cdx" + "?" + urlencode(params)

    print(cdx_url)

    df = pd.read_json(cdx_url)

    headers = df.iloc[0]
    df = pd.DataFrame(df.values[1:], columns=headers)

    df["snapshot_url"] = df["timestamp"].apply(
        lambda ts: "https://web.archive.org/web/" + ts + "/https://doc.iowa.gov/"
    )

    return df["snapshot_url"].to_list()


def scrape_stats_table(url):
    dfs = pd.read_html(url)

    print(url)

    stats_df = dfs[1]
    stats_df = stats_df[["Institution", "Current Count"]]
    stats_df = stats_df[:-2]
    stats_df.loc[
        stats_df["Institution"] == "Forensic Psychiatric Hospital", "Institution"
    ] = "Oakdale - Forensic Psychiatric Hospital"
    stats_df.loc[stats_df["Institution"] == "Minimum", "Institution"] = "Newton-Minimum"
    stats_df.loc[
        stats_df["Institution"] == "Minimum Live-Out", "Institution"
    ] = "Mitchellville - Minimum Live-Out"
    stats_df["url"] = url

    print(stats_df)

    return stats_df


def main():
    snapshot_urls = get_snapshot_urls()

    print("Found {} snapshots".format(len(snapshot_urls)))

    result_dfs = []

    for url in snapshot_urls:
        try:
            stats_df = scrape_stats_table(url)
        except KeyboardInterrupt:
            sys.exit(1)
        except:
            time.sleep(5)
            try:
                stats_df = scrape_stats_table(url)
            except KeyboardInterrupt:
                sys.exit(1)
            except:
                continue

        result_dfs.append(stats_df)

    result_df = pd.concat(result_dfs)
    result_df.to_csv("iowa_doc.csv", index=False)


if __name__ == "__main__":
    main()

We don’t store the timestamp directly in the scraped data as we can trivially recover it from the URL:

from datetime import datetime

import pandas as pd

def convert_timestamp(timestamp):
    return datetime(year=int(timestamp[0:4]), 
                    month=int(timestamp[4:6]), 
                    day=int(timestamp[6:8]), 
                    hour=int(timestamp[8:10]), 
                    minute=int(timestamp[10:12]), 
                    second=int(timestamp[12:14]))

df = pd.read_csv('iowa_doc.csv')
df['timestamp'] = df['url'].apply(lambda url: url.split('/')[4])
df['timestamp'] = df['timestamp'].apply(convert_timestamp)

Some open source projects relying on Internet Archive API:

gau - a recon tool to fetch known URLs from several sources, including Internet Archive.
Bellingcat’s wayback-google-analytics is OSINT tool to dig up Google Analytics identifiers for finding associations between websites.

Obscure corners of a standard macOS installation

Sat, 11 Nov 2023 00:00:00 +0000

Apache httpd

A little known fact is that macOS ships with Apache httpd. You can launch it by running sudo apachectl start. Configuration files are available in /etc/apache2 directory and default DocumentRoot is /Library/WebServer/Documents. Note however that macOS no longer ships PHP/Ruby/Perl/etc. so you will need to install a scripting language separately to do any kind of dynamic web development.

Screenshot 1

macOS has built-in text-to-speech capabilities. To make your Mac speak out some text, pipe it to the say(1) program:

$ echo "colorless green ideas sleep furiously" | say

You can also pass the text as via CLI arguments:

$ say "hello world"

Running it with -v "?" prints a list of voices available. Multiple languages are supported and there are some fun novelty voices. You can choose a voice by passing voice name through -v:

$ say "hello world" -v Bells

say(1) program can be used as TTS component in your shell scripts, cronjobs or other automations.

For most of macOS GUI programs you can highlight some text on the screen and make it spoken through Edit -> Speech -> Start Speaking.

screen(1)

There’s no tmux in a standard macOS install, but older equivalent of tmux - GNU screen - is shipped with macOS. It will be less convenient, but it still gives you an option to make CLI tools run in the background without keeping an active terminal session. Like tmux, it supports terminal session multiplexing with pseudo-window manager features like splitting the terminal into multiple panes. Read the screen(1) manpage for further information.

Grapher

Grapher is a simple GUI tool to draw 2D and 3D graphs of mathematical functions. It supports various coordinate systems such as classical, spheral, logarithmic, cylindrical. Both regular and differential equations are supported.

Screenshot 2

Digital Colour Meter

Digital Colour Meter is a GUI app to measure the exact RGB color values of the pixel under the cursor. This is helpful for GUI or frontend programming tasks that require reproducing the exact colors from the design images. Command-L makes it (un)lock on a point on the screen.

Screenshot 3

Font Book

Font Book is a viewer for all the fonts installed on your macOS system that also allows installing optional or custom fonts.

Screenshot 4

/usr/share/calendar and calendar(1)

Modern macOS is Unix system (officially and ancestrally). Many technical details from BSD systems can be found in macOS. One example is /usr/share/calendar directory that has human and machine readable files listing various notable dates, such as national holidays:

$ ls -lh
total 248
-rw-r--r--  1 root  wheel   483B Sep  2 10:35 calendar.all
-rw-r--r--  1 root  wheel   1.4K Sep  2 10:35 calendar.australia
-rw-r--r--  1 root  wheel    13K Sep  2 10:35 calendar.birthday
-rw-r--r--  1 root  wheel   1.0K Sep  2 10:35 calendar.christian
-rw-r--r--  1 root  wheel   3.2K Sep  2 10:35 calendar.computer
-rw-r--r--  1 root  wheel   269B Sep  2 10:35 calendar.croatian
-rw-r--r--  1 root  wheel   1.7K Sep  2 10:35 calendar.dutch
-rw-r--r--  1 root  wheel    22K Sep  2 10:35 calendar.freebsd
-rw-r--r--  1 root  wheel   265B Sep  2 10:35 calendar.french
-rw-r--r--  1 root  wheel   261B Sep  2 10:35 calendar.german
-rw-r--r--  1 root  wheel    25K Sep  2 10:35 calendar.history
-rw-r--r--  1 root  wheel    21K Sep  2 10:35 calendar.holiday
-rw-r--r--  1 root  wheel   280B Sep  2 10:35 calendar.hungarian
-rw-r--r--  1 root  wheel   6.5K Sep  2 10:35 calendar.judaic
-rw-r--r--  1 root  wheel   1.4K Sep  2 10:35 calendar.lotr
-rw-r--r--  1 root  wheel    13K Sep  2 10:35 calendar.music
-rw-r--r--  1 root  wheel   674B Sep  2 10:35 calendar.newzealand
-rw-r--r--  1 root  wheel   264B Sep  2 10:35 calendar.russian
-rw-r--r--  1 root  wheel   843B Sep  2 10:35 calendar.southafrica
-rw-r--r--  1 root  wheel   269B Sep  2 10:35 calendar.ukrainian
-rw-r--r--  1 root  wheel   1.4K Sep  2 10:35 calendar.usholiday
-rw-r--r--  1 root  wheel   469B Sep  2 10:35 calendar.world
drwxr-xr-x  3 root  wheel    96B Sep  2 10:35 de_AT.ISO_8859-15
drwxr-xr-x  9 root  wheel   288B Sep  2 10:35 de_DE.ISO8859-1
drwxr-xr-x  2 root  wheel    64B Sep  2 10:35 de_DE.ISO8859-15
drwxr-xr-x  7 root  wheel   224B Sep  2 10:35 fr_FR.ISO8859-1
drwxr-xr-x  2 root  wheel    64B Sep  2 10:35 fr_FR.ISO8859-15
drwxr-xr-x  4 root  wheel   128B Sep  2 10:35 hr_HR.ISO8859-2
drwxr-xr-x  5 root  wheel   160B Sep  2 10:35 hu_HU.ISO8859-2
drwxr-xr-x  9 root  wheel   288B Sep  2 10:35 ru_RU.KOI8-R
drwxr-xr-x  6 root  wheel   192B Sep  2 10:35 uk_UA.KOI8-U

For example, calendar.computer lists notable dates from computer history:

$ head -n 20 /usr/share/calendar/calendar.computer
/*
 * Computer
 *
 * $FreeBSD: src/usr.bin/calendar/calendars/calendar.computer,v 1.11 2007/09/07 03:23:06 edwin Exp $
 */

#ifndef _calendar_computer_
#define _calendar_computer_

01/01	AT&T officially divests its local Bell companies, 1984
01/01	The Epoch (Time 0 for UNIX systems, Midnight GMT, 1970)
01/03	Apple Computer founded, 1977
01/08	American Telephone and Telegraph loses antitrust case, 1982
01/08	Herman Hollerith patents first data processing computer, 1889
01/08	Justice Dept. drops IBM suit, 1982
01/10	First CDC 1604 delivered to Navy, 1960
01/16	Set uid bit patent issued, to Dennis Ritchie, 1979
01/17	Justice Dept. begins IBM anti-trust suit, 1969 (drops it, January 8, 1982)
01/24	DG Nova introduced, 1969
01/25	First U.S. meeting of ALGOL definition committee, 1958

There is also calendar(1) - a CLI tool to help you remember some notable dates based on the calendar file (the following output is from 2023 November 10):

$ calendar -f /usr/share/calendar/calendar.usholiday 
Nov 11 	Veterans' Day
$ calendar -f /usr/share/calendar/calendar.world    
Nov 10 	Greg Lake is born in Bournemouth, England, 1948
Nov 10*	Parshas Toldos
Nov 10 	King's Birthday in Bhutan
Nov 10 	Henry Stanley asks David Livingston, "Dr. Livingston, I presume?", 1871
Nov 10 	Cpt. Wirz, commandant of Andersonville Prison hanged, 1865
Nov 10 	41 Women arrested in suffragette demonstrations near White House, 1917
Nov 10 	Soviet President Leonid Brezhnev dies at age 75, 1982
Nov 10 	Martin Luther born in Eisleben, Germany, 1483
Nov 11*	Rosh Chodesh Kislev (Beginning of the month of Kislev)
Nov 11 	Republic Day in Maldives
Nov 11 	Remembrance Day in Canada
Nov 11 	Independence of Cartagena in Colombia
Nov 11 	Independence Day in Angola
Nov 11 	Angola gains independence from Portugal, 1975
Nov 11 	Washington becomes the 42nd state, 1889
Nov 11 	Kurt Vonnegut, Jr, born in Indianapolis, 1922
Nov 12 	Neil Young is born in Toronto, 1945
Nov 12 	USA first exports oil to Europe, 1861
Nov 12 	Dr. Sun Yat-sen's Birthday in Taiwan
Nov 13 	Paul Simon born, 1942
Nov 13 	St. Augustine of Hippo born in Numidia, Algeria, 354
Nov 13 	Robert Louis Stevenson born, 1850

pbcopy(1) and pbpaste(1)

These two tools integrate macOS pasteboard into terminal workflows. Piping text into pbcopy copies it and running pbpaste outputs text from pasteboard into standard output:

$ echo "crypto stands for cryptography" | pbcopy
$ pbpaste
crypto stands for cryptography

networkQuality

networkQuality is CLI tool to measure UL/DL bandwidth and latency of your network connection.

Scraping data from Google Places API

Sun, 05 Nov 2023 00:00:00 +0000

Google Maps Platform contains large amount of POI (Point Of Interest) data that one might want to scrape for purposes such as lead generation, real estate OSINT, academic research and so on. However Google Maps web/mobile apps are highly complex pieces of software and may prove difficult to reverse engineer for data extraction. Once could be doing some browser automation for scraping, but the very same POI data can be extracted through scraping the official Google Places API that is free to use at small scale (Google provides some free monthly credits) and is billed at pay-as-you-go model at non-trivial volumes. We will go through some basic examples of how this data can be scraped with Python. If you retain the data for your own purposes this still violates the platform terms, but it’s pretty much the least shady and most convenient way to extract some of the data from Google Maps Platform as it relies on the official API.

Let us review what we have here to work with:

Place Search functionality is implemented via three endpoints:
- Find Place (/api/place/findplacefromtext/) to search by place name, address, POI category or phone number. Fuzzy matching is applied when searching.
- Nearby Search (/api/place/nearbysearch) to search for places around given coordinates with POI filtering criteria being applied. Can be used to autocomplete search queries in mobile or web apps.
- Text Search (/api/place/textsearch) to search for places by arbitrary search queries. Can be given location and radius to bias (but not filter) search results.
Place Details endpoint (/api/place/details/) takes place_id value from Place Search result and provides more details about the POI including reviews, opening hours, photos and so on. One can apply field masks to skip on some fields to improve cost and performance.
Place Photos endpoint (/api/place/photo) takes a photo_reference from Place Search or Place Details response and gives you a photo rendered to given size preference.

In addition to that, there’s two helper endpoints to assist searches via autocompletion.

At the time of writing Google has launched a set of next generation APIs that you may need to enable via Google Cloud console and switch to if the old API gets deprecated.

For a complete list of data fields one can get, see Place Data Fields page. Note that business emails are not provided via Places API, but phone numbers and website URLs are. If API scraping is done for lead generation purposes, the gap could be bridged by doing opportunistic scraping of company websites or by performing data enrichment via services like Hunter or PDL.

Place Search endpoints have one limitation that must be noted and addressed. One search query can return at most 60 results (3 pages, 20 results each). That may pose a problem if we want to collect non-trivial amounts of data. The solution to this is to compute a grid of locations and run many search queries to cover the territory.

But first we need to get the API key from Google. To proceed, you need a Google Cloud account with billing setup completed. Follow the instructions on these two Google Developer portal pages:

Now we have the API key. We can verify that everything works by doing a single request with curl:

$ curl -s "https://maps.googleapis.com/maps/api/place/details/json?place_id=ChIJj61dQgK6j4AR4GeTYWZsKWw&fields=id,name,international_phone_number&key=[REDACTED]" | jq
{
  "html_attributions": [],
  "result": {
    "international_phone_number": "+1 650-253-0000",
    "name": "Googleplex"
  },
  "status": "OK"
}

We will use the official Google Maps Python module to benefit from the Pythonic API wrapper it provides. Furthermore, we will use the haversine module to help us compute a grid (since Earth is not flat the geospatial software community uses Haversine formula that this module implements for computing distances between points on the globe).

The Python script is as follows:

#!/usr/bin/python3

import csv
import configparser
import math
import os
from pprint import pprint

from haversine import haversine
import googlemaps

FIELDNAMES = ["name", "phone", "address", "website", "latitude", "longitude"]


def make_grid(gmaps_client, grid_element_side, location):
    # Search for location and get bounding box.
    # https://developers.google.com/maps/documentation/places/web-service/search-find-place

    resp = gmaps_client.find_place(location, "textquery", language="en")
    if resp.get("candidates") is None or len(resp.get("candidates")) == 0:
        print("Error: Location not found!")
        sys.exit(1)

    place_id = resp["candidates"][0]["place_id"]

    resp2 = gmaps_client.place(place_id, language="en")

    viewport = resp2.get("result", dict()).get("geometry", dict()).get("viewport")
    if viewport is None:
        print("Error: viewport not found!")
        sys.exit(2)

    from_longitude = viewport.get("southwest").get("lng")
    from_latitude = viewport.get("southwest").get("lat")
    to_longitude = viewport.get("northeast").get("lng")
    to_latitude = viewport.get("northeast").get("lat")

    print("Longitude range: {} to {}".format(from_longitude, to_longitude))
    print("Latitude range: {} to {}".format(from_latitude, to_latitude))

    horiz_len_km = haversine(
        (from_latitude, from_longitude), (from_latitude, to_longitude)
    )
    vert_len_km = haversine(
        (from_latitude, from_longitude), (to_latitude, to_longitude)
    )

    n_horiz = math.ceil(horiz_len_km / grid_element_side)
    n_vert = math.ceil(vert_len_km / grid_element_side)

    grid_elements = []

    latitude_step = (to_latitude - from_latitude) / n_vert
    longitude_step = (to_longitude - from_longitude) / n_horiz

    for i in range(n_horiz):
        for j in range(n_vert):
            south_lat = from_latitude + j * latitude_step

            north_lat = south_lat + latitude_step
            north_lat = min(north_lat, to_latitude)

            west_lng = from_longitude + i * longitude_step
            east_lng = west_lng + longitude_step
            east_lng = min(east_lng, to_longitude)

            grid_elements.append(
                "rectangle:{},{}|{},{}".format(south_lat, west_lng, north_lat, east_lng)
            )

    pprint(grid_elements)

    return grid_elements, from_latitude, to_latitude, from_longitude, to_longitude


def main():
    os.chdir(os.path.dirname(os.path.realpath(__file__)))

    config = configparser.ConfigParser()
    config.read("gplaces.ini")

    google_api_key = config["Google"]["APIKey"]
    grid_element_side = config["Google"]["GridElementSideKM"]
    grid_element_side = float(grid_element_side)

    out_f = open("gplaces.csv", "w", encoding="utf-8")

    gmaps_client = googlemaps.Client(key=google_api_key)

    csv_writer = csv.DictWriter(out_f, fieldnames=FIELDNAMES, lineterminator="\n")
    csv_writer.writeheader()

    location = input("Location: ")
    query = input("Search for: ")

    grid_elements, from_latitude, to_latitude, from_longitude, to_longitude = make_grid(
        gmaps_client, grid_element_side, location
    )

    seen_place_ids = set()

    # Iterate across each subregion and perform search with given query in that region
    for location_bias in grid_elements:
        resp = gmaps_client.find_place(
            query,
            "textquery",
            fields=["place_id", "name", "geometry", "formatted_address"],
            location_bias=location_bias,
            language="en",
        )
        # Iterate across search results and get contact info for each of them.
        # https://developers.google.com/maps/documentation/places/web-service/details
        for result in resp.get("candidates", []):
            place_id = result.get("place_id")
            if place_id in seen_place_ids:
                continue

            latitude = result.get("geometry").get("location").get("lat")
            longitude = result.get("geometry").get("location").get("lng")

            if latitude < from_latitude or latitude > to_latitude:
                continue

            if longitude < from_longitude or longitude > to_longitude:
                continue

            seen_place_ids.add(place_id)

            resp2 = gmaps_client.place(
                place_id,
                fields=["international_phone_number", "website"],
                language="en",
            )

            pprint(result)

            row = {
                "name": result.get("name"),
                "phone": resp2.get("result", dict()).get("international_phone_number"),
                "address": result.get("formatted_address"),
                "website": resp2.get("result", dict()).get("website"),
                "latitude": latitude,
                "longitude": longitude,
            }

            pprint(row)
            csv_writer.writerow(row)

    out_f.close()


if __name__ == "__main__":
    main()

When launched, this script asks to things to be entered into standard input: location name and search query to be used with Text Search API endpoint. The API key is found in the gplaces.ini file that is parsed with a configparser module. Location name is used to find the overall territory and retrieve it’s bounding box in make_grid() function, then split the bounding box into list of grid elements by doing some basic geometric computations. The make_grid() function returns not only list of grid elements in a form that can be used for Place Search locationBias parameter, but also the boundaries of the overall bounding box for some sanity checks to be done later.

Once we have that, we can retrieve the POI data - one grid element at a time. To remove duplicates, we keep place_id value in seen_place_ids set and skip duplicate POIs. We also skip POIs with coordinates outside the bounding box (remember that Text Search endpoint supports biasing, but not filtering search results towards a given location). For each result that passes the filtering, we call the Place Details API (via .place() method of API client) to get two additional fields: international_phone_number and website. The resulting scraped data is written to a CSV file.

Extracting data from elasticsearch API

Mon, 16 Oct 2023 00:00:00 +0000

Elasticsearch is open source server software that acts both as database and search engine, based on Apache Lucene library. It can be used to build distributed clusters for indexing and searching Enterprise-level amounts of data. Some sites that present large, searchable, mostly-textual datasets to the end user are based on elasticsearch as their backend data store and have frontend code talking directly to the API of elasticsearch. If the frontend can talk directly to elasticsearch, so can we as web scraper developers with all the benefits the flexible data search engine can bring. A specific example is the website of Carnegie Museum of Art. To see how the trick works, we will be scraping this site today.

If we go to collection.carnegieart.org with web browser we see a list of of various objects being presented with search icon on the upper right corner of the page. Loading this page relies on elasticsearch API, as can be seen in Network tab of Chrome DevTools.

For instance, the following request (represented by curl snippet) fetches the first page worth of data:

curl 'https://collection.carnegieart.org/api/cmoa_objects/_msearch?' \
  -H 'authority: collection.carnegieart.org' \
  -H 'accept: application/json' \
  -H 'accept-language: en-GB,en-US;q=0.9,en;q=0.8' \
  -H 'authorization: Basic Y29sbGVjdGlvbnM6bzQxS0chTW1KUSRBNjY=' \
  -H 'cache-control: no-cache' \
  -H 'content-type: application/x-ndjson' \
  -H 'origin: https://collection.carnegieart.org' \
  -H 'pragma: no-cache' \
  -H 'referer: https://collection.carnegieart.org/' \
  -H 'sec-ch-ua: "Chromium";v="116", "Not)A;Brand";v="24", "Google Chrome";v="116"' \
  -H 'sec-ch-ua-mobile: ?0' \
  -H 'sec-ch-ua-platform: "macOS"' \
  -H 'sec-fetch-dest: empty' \
  -H 'sec-fetch-mode: cors' \
  -H 'sec-fetch-site: same-origin' \
  -H 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36' \
  --data-raw $'{"preference":"results"}\n{"query":{"bool":{"must":[{"bool":{"must":[{"bool":{"must":[{"term":{"images.permitted":true}},{"exists":{"field":"images.filename"}}]}}]}}]}},"size":24,"sort":[{"acquisition_date":{"order":"desc"}}]}\n' \
  --compressed

Screenshot 1 Screenshot 2

Some things are notable here. A content-type header is application/x-ndjson which means Newline-Delimited JSON is provided in the POST request payload. That entails not a single JSON array or object, but a sequence of JSON objects - one per line. That’s because the Multi search API is used to populate the pages. Each line (even the last one) must have a newline at the end. The first line is a search header and provides basic parameters. The second line provides a configuration for search query (pretty-printed for readability):

{
  "query": {
    "bool": {
      "must": [
        {
          "bool": {
            "must": [
              {
                "bool": {
                  "must": [
                    {
                      "term": {
                        "images.permitted": true
                      }
                    },
                    {
                      "exists": {
                        "field": "images.filename"
                      }
                    }
                  ]
                }
              }
            ]
          }
        }
      ]
    }
  },
  "size": 24,
  "sort": [
    {
      "acquisition_date": {
        "order": "desc"
      }
    }
  ]
}

At query key we have some nested predicate that basically requires object images to be available (“Has Images” checkbox is on by default). At size we have a page size and at sort key we have sorting configuration.

If we go to next page we start having another key here - from that enables pagination by search result offset.

A regular JSON payload is available in the payload of response and has a great deal of information on the museum objects. So the good news is that we don’t need to parse any kind of HTML here and can fully rely on pre-structured data that we can access through elasticsearch API. The entire dataset of Carnegie Museum of Art object catalog can be gathered solely by doing API scraping.

Screenshot 3

There is also the authorization header here with Base64-encoded API credentials. This does not add any security as we can trivally decode the encoded part of the header and recover the credentials that frontend code uses:

$ echo "Y29sbGVjdGlvbnM6bzQxS0chTW1KUSRBNjY=" | base64 -d
collections:o41KG!MmJQ$A66

Now, if we uncheck the “Has Image” checkbox the second line in the request payload becomes even simpler:

{
  "query": {
    "match_all": {}
  },
  "size": 24,
  "sort": [
    {
      "acquisition_date": {
        "order": "desc"
      }
    }
  ]
}

Screenshot 4

Now the query covers all the data (over 90 000 records!). But there’s a little problem - the elasticsearch Multi Search API that is used here is not meant to provide all the search results. For that purpose we will be using the Scroll API that is designed to provide a paginated access to search results after the first (regular, not multiple) Search API request is performed. But first let us prepare a requests.Session object with the proper headers:

# elasticsearch API credentials:
USERNAME = "collections"
PASSWORD = "o41KG!MmJQ$A66"


def create_session():
    session = requests.Session()

    session.headers = {
        "authority": "collection.carnegieart.org",
        "accept": "application/json",
        "accept-language": "en-GB,en-US;q=0.9,en;q=0.8",
        "cache-control": "no-cache",
        "content-type": "application/json",
        "origin": "https://collection.carnegieart.org",
        "pragma": "no-cache",
        "referer": "https://collection.carnegieart.org/",
        "sec-ch-ua": '"Chromium";v="116", "Not)A;Brand";v="24", "Google Chrome";v="116"',
        "sec-ch-ua-mobile": "?0",
        "sec-ch-ua-platform": '"macOS"',
        "sec-fetch-dest": "empty",
        "sec-fetch-mode": "cors",
        "sec-fetch-site": "same-origin",
        "user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36",
    }

    session.auth = (USERNAME, PASSWORD)

    return session

session = create_session()

You may notice that the content type we say we’re using is application/json - a regular JSON format, not the multiline variant. That’s because we will be using slightly different APIs than what the site frontend relies on.

When launching the first search request for the first page, we provide a scroll URL parameter with some duration that we want the search results to be retained for between requests:

def scrape(session):
    json_payload = {
        "query": {"match_all": dict()},
        "size": 24,
        "from": 0,
        "sort": [{"acquisition_date": {"order": "desc"}}],
    }

    params = {"scroll": "1m"}

    resp0 = session.post(
        "https://collection.carnegieart.org/api/cmoa_objects/_search",
        json=json_payload,
        params=params,
    )
    print(resp0.url)

    yield from gen_rows_from_response_payload(resp0.json())

Here 1m stands for 1 minute. This value is known as Scroll Window.

This gives us the first page of the dataset AND a _scroll_id field in the JSON response that we can use with Scroll API to get further data:

    scroll_id = resp0.json().get("_scroll_id")

    while scroll_id is not None:
        resp = session.get(
            "https://collection.carnegieart.org/api/_search/scroll/" + scroll_id,
            params=params,
        )
        print(resp.url)

        yield from gen_rows_from_response_payload(resp.json())
        hits = resp.json().get("hits", dict()).get("hits", [])
        if len(hits) == 0:
            break

        scroll_id = resp.json().get("_scroll_id")

The generator function gen_rows_from_response_payload() we use here helps with converting some of the stuff in API response to data rows we can write into CSV file.

The entire scraper script is as follows:

#!/usr/bin/python3

import csv
import json
from urllib.parse import quote

import requests

# elasticsearch API credentials:
USERNAME = "collections"
PASSWORD = "o41KG!MmJQ$A66"


def create_session():
    session = requests.Session()

    session.headers = {
        "authority": "collection.carnegieart.org",
        "accept": "application/json",
        "accept-language": "en-GB,en-US;q=0.9,en;q=0.8",
        "cache-control": "no-cache",
        "content-type": "application/json",
        "origin": "https://collection.carnegieart.org",
        "pragma": "no-cache",
        "referer": "https://collection.carnegieart.org/",
        "sec-ch-ua": '"Chromium";v="116", "Not)A;Brand";v="24", "Google Chrome";v="116"',
        "sec-ch-ua-mobile": "?0",
        "sec-ch-ua-platform": '"macOS"',
        "sec-fetch-dest": "empty",
        "sec-fetch-mode": "cors",
        "sec-fetch-site": "same-origin",
        "user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36",
    }

    session.auth = (USERNAME, PASSWORD)

    return session


def gen_rows_from_response_payload(resp_json):
    hits = resp_json.get("hits", dict()).get("hits", [])

    for hit_dict in hits:
        source_dict = hit_dict.get("_source", dict())

        row = dict()

        row["title"] = source_dict.get("title")
        row["department"] = source_dict.get("department")
        row["category"] = source_dict.get("type")
        row["current_location"] = source_dict.get("current_location")
        row["materials"] = source_dict.get("medium")
        row["technique"] = source_dict.get("medium_description")
        row["from_location"] = source_dict.get("creation_address")
        row["date_description"] = source_dict.get("creation_date")[0]

        try:
            row["year_start"] = int(source_dict.get("creation_date")[0])
            row["year_end"] = int(source_dict.get("creation_date")[-1])
        except:
            pass

        makers = source_dict.get("creators", [])
        if makers is not None and len(makers) > 0:
            maker_names = list(map(lambda m: m.get("label"), makers))
            maker_birth_years = list(map(lambda m: str(m.get("birth", "")), makers))
            maker_death_years = list(map(lambda m: str(m.get("death", "")), makers))
            maker_genders = list(map(lambda m: str(m.get("gender")), makers))

            row["maker_full_name"] = "|".join(maker_names)
            row["maker_birth_year"] = "|".join(maker_birth_years).replace("None", "")
            row["maker_death_year"] = "|".join(maker_death_years).replace("None", "")
            row["maker_gender"] = "|".join(maker_genders).replace("None", "")

        try:
            row["acquired_year"] = source_dict.get("acquisition_date", "").split("-")[
                0
            ]
        except:
            pass

        row["acquired_from"] = source_dict.get("acquisition_method")
        row["accession_number"] = source_dict.get("accession_number")
        row["credit_line"] = source_dict.get("credit_line")

        row["url"] = "https://collection.carnegieart.org/objects/" + source_dict.get(
            "id", ""
        ).replace("cmoa:objects/", "")

        yield row


def scrape(session):
    json_payload = {
        "query": {"match_all": dict()},
        "size": 24,
        "from": 0,
        "sort": [{"acquisition_date": {"order": "desc"}}],
    }

    params = {"scroll": "1m"}

    resp0 = session.post(
        "https://collection.carnegieart.org/api/cmoa_objects/_search",
        json=json_payload,
        params=params,
    )
    print(resp0.url)

    yield from gen_rows_from_response_payload(resp0.json())

    scroll_id = resp0.json().get("_scroll_id")

    while scroll_id is not None:
        resp = session.get(
            "https://collection.carnegieart.org/api/_search/scroll/" + scroll_id,
            params=params,
        )
        print(resp.url)

        yield from gen_rows_from_response_payload(resp.json())
        hits = resp.json().get("hits", dict()).get("hits", [])
        if len(hits) == 0:
            break

        scroll_id = resp.json().get("_scroll_id")


def main():
    session = create_session()

    out_f = open("carnegie.csv", "w", encoding="utf-8")
    csv_writer = None

    for row in scrape(session):
        print(row)
        if csv_writer is None:
            csv_writer = csv.DictWriter(
                out_f, fieldnames=list(row.keys()), lineterminator="\n"
            )
            csv_writer.writeheader()

        csv_writer.writerow(row)


if __name__ == "__main__":
    main()

You may want to run this through automatically rotating DC proxy pool as there seems to be a little IP-based rate-limiting being done on the server.

Since we have access to the API of elasticsearch database, we can use it to develop unofficial/adversarial integrations by either doing the API calls directly as in above example of by using one of the elasticsearch client libraries. There are two official options for Python:

elasticsearch-py for bridging the gap between ES API constructs and Python dictionaries.
elasticsearch-dsl-py for a higher order API that provides a more convenient way to work with database while not straying too far from the JSON-based API.

Setting up mitmproxy with iOS 17.1

Wed, 04 Oct 2023 00:00:00 +0000

mitmproxy is open source, interactive proxy server meant for interception, capture and analysis of network traffic for purposes such as penetration testing, debugging, troubleshooting, reverse engineering and privacy analysis. A common use case of this tool is to explore mobile application communications (typically API calls over HTTPS) for API scraping, automation and security testing. Earlier in the blog I have covered how to set up mitmproxy with mobile device that run iOS 15. Today iOS 17.1 beta 2 is available for end users and thus we will go through the process of setting it up with mitmproxy.

Let us start with installing mitmproxy itself. There are several ways to do it depending on your needs and specifics of dev environment:

On macOS you can install it through Homebrew: brew install mitmproxy.
If you want to develop mitmproxy extensions in Python it can be installed via PIP: pip3 install mitmproxy.
If you want to run it via Docker, the official image is available on Docker hub.
Some Linux distributions (e.g. Debian, Arch, Kali) ship mitmproxy in their package managers.
Windows installer, source tarballs and Linux/macOS binaries are available from the official website.

To make sure you have it installed correctly, run it with --version CLI argument:

$ mitmproxy --version
Mitmproxy: 10.0.0
Python:    3.11.5
OpenSSL:   OpenSSL 3.1.3 19 Sep 2023
Platform:  macOS-13.5.2-arm64-arm-64bit

Now run the mitmproxy TUI program in terminal of your local machine (behind NAT). Doing this on server is possible as well, but not recommended. Either way, you need to make sure that your iPhone can reach the mitmproxy server via the network. A typical setup is having them both in a single WiFi network.

Now you need to find out the IP address of your server:

On macoS run ifconfig and look for IPv4 address under your primary network interface (e.g. en0).
On Linux it will depend on exact system specifics, but you can try using ifconfig if available, or running ip addr or hostname -I.
On Windows it will be available in the output of ipconfig command.

If you have desktop GUI environment running the IP address will be available through network setting app as well.

Let us do the setup on iPhone side. Make sure you are connected to the right network and go to the Settings app.

Screenshot 1

Press Wi-Fi cell in the table view and tap the (i) button next to the name of Wi-Fi network you are connected to.

Screenshot 2

In the next screen go to the bottom of the table and tap “Configure Proxy” cell.

Screenshot 3

Choose “Manual” from the available options and fill the little form here:

In “Server” cell enter the IP address of your computer you have determined earlier.
In “Port” field enter proxy port number - 8080.

Screenshot 4

Press the “Save” button on the top of UI.

To intercept the HTTPS traffic, we need to install an X.509 certificate. Open the Safari (specifically Safari - not some other browser) and navigate to mitm.it. Press the green button under “iOS” heading. Allow the browser to download the file into your system.

Screenshot 5 Screenshot 6

Install the certificate through Settings app: go to General, then to “VPN & Device Management”.

Screenshot 7

You will find “mitmproxy” entry under “Downloaded Profile” label. Tap it and press Install button in the modal view. Since this is a sensitive action, iOS will ask for PIN code and additional verification. If all goes well, a “Verified” label with a checkmark will appear on the GUI.

Screenshot 8

But this is not enough. We installed the certificate, but we also need to trust it. Go back to General section of settings, then go to About and choose “Certificate Trust Settings” at the bottom. Turn on the switch for the mitmproxy certificate.

Screenshot 9

Now let us verify that it works correctly. Go back to Safari and open some website that is protected by TLS (which most modern websites use). You should see HTTPS requests related to the site displayed in mitmproxy TUI.

Screenshot 10 Screenshot 11

If this does not work right away you may want to set your iPhone to airplane mode for a brief period of time and make it reconnect to Wi-Fi network. That used to help in earlier versions of iOS.

If you don’t like the TUI version of mitmproxy you can try mitmweb. It also ships with mitmproxy, but presents data in web interface.

Screenshot 12 Screenshot 13

Another thing that ships with mitmproxy is mitmdump - a tool similar to tcpdump, but based on mitmproxy machinery and made to primarily intercept HTTP(S) traffic:

$ mitmdump
[13:29:03.684] HTTP(S) proxy listening at *:8080.
[13:29:04.825][192.168.0.16:52306] client connect
[13:29:04.883][192.168.0.16:52306] server connect bam.nr-data.net:443 (162.247.243.29:443)
192.168.0.16:52306: POST https://bam.nr-data.net/jserrors/1/NRBR-26d48a12bc762b8fbd7?a=45012625&v=1.242.0&to=ZVdXbUtXXBIHVUNfXlwde1ZLW1MND0xSUmRAWxoT&rst=93147&ck=0&s=25398efdfbe5dc5f&ref=https://www.nike…
                 << 200 OK 24b
[13:29:05.237][192.168.0.16:52307] client connect
[13:29:05.298][192.168.0.16:52307] server connect www.nike.com:443 (2.18.32.173:443)
192.168.0.16:52307: POST https://www.nike.com/Fj-Dfn/Fyj/7WK/PLInpt0t/7pEchtzQph7ruY/STdCAQ/SVJ3bWdc/bUgB HTTP/2.0
        << HTTP/2.0 201 Created 18b
192.168.0.16:52307: POST https://www.nike.com/Fj-Dfn/Fyj/7WK/PLInpt0t/7pEchtzQph7ruY/STdCAQ/SVJ3bWdc/bUgB HTTP/2.0
        << HTTP/2.0 201 Created 18b
192.168.0.16:52307: POST https://www.nike.com/Fj-Dfn/Fyj/7WK/PLInpt0t/7pEchtzQph7ruY/STdCAQ/SVJ3bWdc/bUgB HTTP/2.0
 << stream reset by client (CANCEL)
192.168.0.16:52307: GET https://www.nike.com/w/air-force-1-shoes-5sj3yzy7ok HTTP/2.0
        << HTTP/2.0 200 OK 115k
[13:29:06.554][192.168.0.16:52308] client connect
[13:29:06.556][192.168.0.16:52309] client connect
...

When you no longer want the traffic to be intercepted, go back to “Configure Proxy” table in iPhone settings and disable the proxying there.

Parsing HTML documents with Beautiful Soup

Fri, 22 Sep 2023 00:00:00 +0000

Beautiful Soup (also known as bs4) is a Python module for parsing HTML (and also XML) documents. It is commonly used in web scraper development. Beautiful Soup takes an HTML string and parses it into a tree structure that reflects the structure of DOM tree, but has properties and methods related to accessing data and running queries.

We can install it via PIP as beautifulsoup4. Furthermore, some operating systems ship Beautiful Soup in their package managers. For example, there’s python3-bs4 package on Debian APT.

Beautiful Soup does not perform parsing on it’s own. Instead, it uses a lower level HTML/XML parsing library and converts the results into it’s own data model. As of September 2023, there are following options for the HTML parsing library:

Python’s vanilla HTML parser (html.parser) - middling performance and average tolerance to malformed HTML.
lxml - Python wrapper around libxml2 C library - very good performance, quite lenient for bad HTML.
html5lib - Pure Python HTML5 parser that is slow, but can deal with some pretty messed up pages.

Since Beautiful Soup is an abstraction over these HTML/XML parsers we have some flexibility when choosing the underlying parser while keeping the same data model.

Let us explore how to use Beautiful Soup for web scraping. We start by using the lxml parser on a simple HTML document.

$ python3
Python 3.11.5 (main, Aug 24 2023, 15:09:45) [Clang 14.0.3 (clang-1403.0.22.14.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from bs4 import BeautifulSoup
>>> soup = BeautifulSoup(html_str, 'lxml')
>>> soup
<html><head><title>Title!</title></head><body><h1>The Title</h1></body></html>
>>> type(soup)
<class 'bs4.BeautifulSoup'>

We can regenerate the HTML document in a pretty-printed form:

>>> print(soup.prettify())
<html>
 <head>
  <title>
   Title!
  </title>
 </head>
 <body>
  <h1>
   The Title
  </h1>
 </body>
</html>

We can use object properties to traverse across the tree structure and extract stuff we may want to parse:

>>> soup.title
<title>Title!</title>
>>> soup.title.text
'Title!'
>>> soup.title.name
'title'
>>> soup.body.h1.text
'The Title'

To provide an example on how to run queries, let us have a bigger HTML document to parse:

<html>
    <head>
        <title>Product list</title>
    </head>
    <body>
        <table border="1">
            <thead>
                <tr>
                    <th>Product name</th>
                    <th>Price</th>
                    <th>URL</th>
                </tr>
            </thead>
            <tbody>
                <tr>
                    <td id="productname" class="productname">Jordan 4 Retro</td>
                    <td id="price">$269</td>
                    <td><a id="pdpurl" href="https://stockx.com/air-jordan-4-retro-red-cement">link</a></td>
                </tr>
                <tr>
                    <td id="productname" class="productname">adidas Yeezy Boost 350 V2 Static (Non-Reflective) (2018/2023)</td>
                    <td id="price">$195</td>
                    <td><a id="pdpurl" href="https://stockx.com/adidas-yeezy-boost-350-v2-static">link</a></td>
                </tr>
            </tbody>
        </table>
    </body>
</html>

Now we can use find() method to find a single result or find_all() to find multiple results:

>>> in_f = open("test.html", "r")
>>> html_str = in_f.read()
>>> in_f.close()
>>> soup = BeautifulSoup(html_str, "lxml")
>>> soup.find('title')
<title>Product list</title>
>>> soup.find_all(class_='productname')
[<td class="productname" id="productname">Jordan 4 Retro</td>, <td class="productname" id="productname">adidas Yeezy Boost 350 V2 Static (Non-Reflective) (2018/2023)</td>]
>>> soup.find_all(id='price')
[<td id="price">$269</td>, <td id="price">$195</td>]
>>> soup.find_all(attrs={'class':'productname', 'id':'productname'})
[<td class="productname" id="productname">Jordan 4 Retro</td>, <td class="productname" id="productname">adidas Yeezy Boost 350 V2 Static (Non-Reflective) (2018/2023)</td>]

To extract text recursively from a (sub)tree we can use get_text() method:

>>> print(soup.get_text())


Product list





Product name
Price
URL




Jordan 4 Retro
$269
link


adidas Yeezy Boost 350 V2 Static (Non-Reflective) (2018/2023)
$195
link

Let us fetch a more complex page for more advanced example (Bright Data Web Unlocker is used here):

>>> import requests
>>> proxy_url = "[REDACTED]"
>>> resp = requests.get("https://clutch.co/directory/iphone-application-developers", proxies={'https': proxy_url})
>>> resp = requests.get("https://clutch.co/directory/iphone-application-developers", proxies={'https': proxy_url}, verify=False)
/opt/homebrew/lib/python3.11/site-packages/urllib3/connectionpool.py:1045: InsecureRequestWarning: Unverified HTTPS request is being made to host 'brd.superproxy.io'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
  warnings.warn(
>>> resp
<Response [200]>

By inspecting the page we see that list items are children of <ul> element with class directory-list.

Screenshot 1 Screenshot 2 Screenshot 3

Now, let us try to parse it and extract some data based on structure we see when inspecting the page:

>>> soup = BeautifulSoup(resp.text, 'html.parser')
>>> ul = soup.find('ul', class_='directory-list')
>>> list_items = ul.find_all('li')
>>> li = list_items[0]
>>> title = li.find(attrs={'data-link_text': 'Profile Title'})
>>> title
<a class="company_title directory_profile" data-link_text="Profile Title" href="/profile/naked-development" target="_blank">
                            Naked Development
                        </a>
>>> title = li.find(attrs={'data-link_text': 'Profile Title'}).text
>>> title
'\n                            Naked Development\n                        '
>>> title = title.strip()
>>> title
'Naked Development'
>>> li.find(class_="company_info__wrap").text
'Creative Development Agency'
>>> li.find(attrs={'data-link_text': 'Profile Title'}).attrs['href']
'/profile/naked-development'
>>> li.find(class_="website-link__item").attrs['href']
'https://app.nakeddev.com/application?utm_source=clutch.co&utm_medium=referral&utm_campaign=directory-iphone-application-developers'

Happy scraping!

Solving a simple JS challenge with sandboxing

Fri, 15 Sep 2023 00:00:00 +0000

Many antibot solutions involve client-side JavaScript challenges - performing randomised computation, probing browser environment and so on. That may pose an obstacle for web scraper and automation bot development. In some cases this involves heavily obfuscated or even virtualised code that is hard to reverse engineer. Ideally we want to fully understand and reproduce the challenge in our code, but that is not always easy or worth the time and effort. A simpler way to deal with JS challenges is technique known as sandboxing - creating an artificial, isolated environment to run the challenge code with little to no modifications and extracting the outputs from the code (e.g. HTTP headers or cookies). In this post we will go through a simple example of this technique.

For the sake of example, we need a sample of JS challenge code. Some sites, such as OpenCorporates provide us with a simple one to play with:

$ curl https://opencorporates.com
<html><head><script type="text/javascript"><!--
function leastFactor(n) {
 if (isNaN(n) || !isFinite(n)) return NaN;
 if (typeof phantom !== 'undefined') return 'phantom';
 if (typeof module !== 'undefined' && module.exports) return 'node';
 if (n==0) return 0;
 if (n%1 || n*n<2) return 1;
 if (n%2==0) return 2;
 if (n%3==0) return 3;
 if (n%5==0) return 5;
 var m=Math.sqrt(n);
 for (var i=7;i<=m;i+=30) {
  if (n%i==0)      return i;
  if (n%(i+4)==0)  return i+4;
  if (n%(i+6)==0)  return i+6;
  if (n%(i+10)==0) return i+10;
  if (n%(i+12)==0) return i+12;
  if (n%(i+16)==0) return i+16;
  if (n%(i+22)==0) return i+22;
  if (n%(i+24)==0) return i+24;
 }
 return n;
}
function go() {
 var p=1612862479081; var s=3718246712; var n;
if ((s >> 12) & 1)/*
*13;
*/p+=120385074*15;
else 
p-=/*
else p-=
*/126371223* 13;	if ((s >> 6) & 1)/*
*13;
*/p+=159919203* 9;
else 	p-=
142123288*	7;if ((s >> 11) & 1)/*
*13;
*/p+=135381145*/*
p+= */12;
else /*
*13;
*/p-=/*
else p-=
*/79540378* 12;	if ((s >> 13) & 1) p+=/*
p+= */86227786*
14; else  p-=/* 120886108*
*/78525417*
14;/*
p+= */if ((s >> 14) & 1) p+=/* 120886108*
*/104707476*/*
else p-=
*/17;/*
else p-=
*/else 	p-= 23237263*
15;
 p-=3141693876;
 n=leastFactor(p);
{ document.cookie="KEY="+n+"*"+p/n+":"+s+":2460963898:1;path=/;";
  document.location.reload(true); }
}
//--></script></head>
<body onload="go()">
Loading...
</body>
</html>

The HTML document calls the go() function when it is loaded. The go() function calls leastFactor() to perform some checks and computations. There are some funny comments meant to throw off people who are using regular expresions to clean up code, but we can get rid of them at AST level. Let us copy-paste the JS code into AST Explorer and apply the following Babel transform:

export default function (babel) {
  const { types: t } = babel;
  
  return {
    name: "ast-transform", // not required
    visitor: {
      "Program|IfStatement|Expression|ExpressionStatement"(path) {
        t.removeComments(path.node);
      }
    }
  };
}

This yields the following code with junk comments removed:

function leastFactor(n) {
  if (isNaN(n) || !isFinite(n)) return NaN;
  if (typeof phantom !== 'undefined') return 'phantom';
  if (typeof module !== 'undefined' && module.exports) return 'node';
  if (n == 0) return 0;
  if (n % 1 || n * n < 2) return 1;
  if (n % 2 == 0) return 2;
  if (n % 3 == 0) return 3;
  if (n % 5 == 0) return 5;
  var m = Math.sqrt(n);

  for (var i = 7; i <= m; i += 30) {
    if (n % i == 0) return i;
    if (n % (i + 4) == 0) return i + 4;
    if (n % (i + 6) == 0) return i + 6;
    if (n % (i + 10) == 0) return i + 10;
    if (n % (i + 12) == 0) return i + 12;
    if (n % (i + 16) == 0) return i + 16;
    if (n % (i + 22) == 0) return i + 22;
    if (n % (i + 24) == 0) return i + 24;
  }

  return n;
}

function go() {
  var p = 1612862479081;
  var s = 3718246712;
  var n;
  if (s >> 12 & 1) p += 120385074 * 15;else p -= 126371223 * 13;
  if (s >> 6 & 1) p += 159919203 * 9;else p -= 142123288 * 7;
  if (s >> 11 & 1) p += 135381145 * 12;else p -= 79540378 * 12;
  if (s >> 13 & 1) p += 86227786 * 14;else p -= 78525417 * 14;
  if (s >> 14 & 1) p += 104707476 * 17;else p -= 23237263 * 15;
  p -= 3141693876;
  n = leastFactor(p);
  {
    document.cookie = "KEY=" + n + "*" + p / n + ":" + s + ":2460963898:1;path=/;";
    document.location.reload(true);
  }
}

Screenshot

Aside from some random computations with random numbers, there are two things to notice here. First are checks for symptoms of automation on the top of leastFactor() function:

  if (typeof phantom !== 'undefined') return 'phantom';
  if (typeof module !== 'undefined' && module.exports) return 'node';

We have one check for now obsolete PhantomJS browser automation software and another check for NodeJS environment. If any of these fail the the main computation is not performed and correct result is not achieved.

Another thing to pay attention to is the following two statements at the bottom of code:

    document.cookie = "KEY=" + n + "*" + p / n + ":" + s + ":2460963898:1;path=/;";
    document.location.reload(true);

The first one sets the cookie based on initial hardcoded numbers and return value of leastFactor() function. Second one forces page reload, so that web server could see the cookie and give the proper page instead of JS challenge we got here.

So if we want to scrape the site we need to manufacture the cookie based on JS challenge code we’re getting without triggering checks for traces of automation. Is it trivial to pass the first check as PhantomJS is dead as of 2023. We could tamper with the code bit more and remove the check for NodeJS modules, but this would entail a bit of cheating, as generally JS challenges are obfuscated in more advanced way than what we see here.

Some options to create sandbox environment are:

NodeJS vm API
vm2 module from NPM (recently discontinued)
isolated-vm NPM module
JSDOM - a lightweight emulator for browser-like JS environment.
Using an actual browser and controlling it programmatically.

We will keep it simple here and show a simple example based on the first option. We save the challenge code into in.js. The following few lines of JS will read it and execute the challenge code in a sandbox environment to get the cookie:

const fs = require("fs");
const vm = require("vm");

let js = fs.readFileSync("in.js", "utf-8");

const script = new vm.Script(js);
const mapperContext = { window: {}, document: {
  location: {
    reload: function(){}
  }
}};

const sandbox = vm.createContext(mapperContext);
script.runInContext(sandbox);
vm.runInContext('go();', sandbox);

console.log(sandbox.document.cookie);

We save this code into sandbox.js file.

Note that when creating a VM context (a sandbox) we provide an object called mapperContext with some things that JS code expects to be there stubbed out. For instance, to prevent the challenge code from crashing we provide an empty implementation for document.location.reload() function. This ensures that prerequisites for challenge code execution are met.

Next, there’s script.runInContext(sandbox);. This statement “runs” the code in VM, but since the original code contains only the two function declarations without any call expression, we get it going in next line by calling go() function in the sandbox environment. Lastly, we retrieve and print the document.cookie value from the sandbox.

It turned out modifying the original code in any way was not necessary and that it can be sandboxed as-is.

The following Bash script shows the entire steps to get the challenge, solve it with sandboxing and using the cookie value to retrieve a proper page:

#!/bin/bash

set -x

curl "https://opencorporates.com" | \
    xmllint --html --xpath '//script/text()' - | \
    tail +2 > in.js # Deleting "<![CDATA[<!--" line

curl "https://opencorporates.com" -H "cookie: $(node sandbox.js)"

More advanced examples of JS sandboxing from and outside grayhat space are:

FlareSolverr - a CloudFlare solver API server that relies on undetected-chromedriver to solve CF challenges in modified browser environment.
Google allows running some subset of JS language in Google Tag Manager templates to enable further customisation. See the Sandboxed JavaScript page.

Abusing GOAT mobile API for data scraping

Mon, 14 Aug 2023 00:00:00 +0000

GOAT is an online platform for retail sale and reselling of certain consumer products - primarily sneakers, apparel and electronics. It is estimated to have about 50 million active users that include some one million sellers. When there’s a major userbase, there’s a potential to extract monetary benefit by scraping the data and/or running some automations. But 1661 Inc., a company that is developing GOAT don’t want us to do that. GOAT is one of the websites that proactively fight automated traffic. For example, merely trying load the front page in Scrapy shell gives us a Cloudflare error.

Screenshot 1

So we have to proceed differently. One trick from grayhat automation arsenal is to MITM mobile app traffic to work out the exact API calls being done against the backend system, then replay them programmatically. Our example will be a simple API scraping to extract data.

We will be using mitmproxy to intercept GOAT API traffic. The setup in question is described in earlier post, except that a newer iOS version is used on the iPhone.

When we run the app and press “Start browsing” button we see quite a bit of traffic in mitmproxy TUI.

Screenshot 2

Some things to notice here:

Early in the app lifecycle it sends a request to PerimeterX API. This means that PerimeterX is also used to fight automation.

Screenshot 3

Some requests have x-px-authorization and x-px-original-token headers that are related to PerimeterX. Furthermore, __cf_bm cookie is present on some requests.
There’s authorization header with empty value. We expect this value to be set when app is being used after logging in.

Screenshot 4

Some requests go to goat.cnstrc.com domain and are not covered by PX/CF.

Screenshot 5

Let us start with the simple stuff first. Product data scraping is a common thing that web scraper developers do. So let’s do that, but through the mobile API. On the mobile GUI we access the Search tab, type in a search query, press z to clean the flows captured by mitmproxy and then press the search button.

Screenshot 5 Screesnhot 6

We see one request to goat.cnstrc.com and some more requests to images.goat.com.

Screenshot 7

The first one is of primary interest to us, as it seems to return some subset of product data in the JSON payload. Last component of the URL path is the search query that was typed into the app.

We want to export this request as curl command. We position the curson on it in the flow list view, press e and choose 1) curl from list of options in the little dialog box. Then we type the file path (e.g. search.curl) into bottom part of user interface.

curl -H 'accept: */*' --compressed -H 'user-agent: GOAT/19 CFNetwork/1410.0.3 Darwin/22.6.0' -H 'accept-language: en-GB,en;q=0.9' -H 'x-emb-id: 7E2DEE62833C40A0B733085027D1A5BC' 'https://goat.cnstrc.com/search/gtx?key=key_XT7bjdbvjgECO5d8&page=1&fmt_options%5Bhidden_fields%5D=gp_lowest_price_cents_2&fmt_options%5Bhidden_facets%5D=gp_lowest_price_cents_2&fmt_options%5Bhidden_fields%5D=gp_instant_ship_lowest_price_cents_2&fmt_options%5Bhidden_facets%5D=gp_instant_ship_lowest_price_cents_2&features%5Bdisplay_variations%5D=true&feature_variants%5Bdisplay_variations%5D=matched&variations_map=%7B%22dtype%22:%22object%22,%22group_by%22:%5B%7B%22name%22:%22product_condition%22,%22field%22:%22data.product_condition%22%7D,%7B%22name%22:%22box_condition%22,%22field%22:%22data.box_condition%22%7D%5D,%22values%22:%7B%22min_regional_instant_ship_price%22:%7B%22field%22:%22data.gp_instant_ship_lowest_price_cents_2%22,%22aggregation%22:%22min%22%7D,%22min_regional_price%22:%7B%22field%22:%22data.gp_lowest_price_cents_2%22,%22aggregation%22:%22min%22%7D%7D%7D'

By putting this into curlconverter we get the following Python snippet:

import requests

headers = {
    'accept': '*/*',
    'user-agent': 'GOAT/19 CFNetwork/1410.0.3 Darwin/22.6.0',
    'accept-language': 'en-GB,en;q=0.9',
    'x-emb-id': '7E2DEE62833C40A0B733085027D1A5BC',
}

response = requests.get(
    'https://goat.cnstrc.com/search/gtx?key=key_XT7bjdbvjgECO5d8&page=1&fmt_options%5Bhidden_fields%5D=gp_lowest_price_cents_2&fmt_options%5Bhidden_facets%5D=gp_lowest_price_cents_2&fmt_options%5Bhidden_fields%5D=gp_instant_ship_lowest_price_cents_2&fmt_options%5Bhidden_facets%5D=gp_instant_ship_lowest_price_cents_2&features%5Bdisplay_variations%5D=true&feature_variants%5Bdisplay_variations%5D=matched&variations_map=%7B%22dtype%22:%22object%22,%22group_by%22:%5B%7B%22name%22:%22product_condition%22,%22field%22:%22data.product_condition%22%7D,%7B%22name%22:%22box_condition%22,%22field%22:%22data.box_condition%22%7D%5D,%22values%22:%7B%22min_regional_instant_ship_price%22:%7B%22field%22:%22data.gp_instant_ship_lowest_price_cents_2%22,%22aggregation%22:%22min%22%7D,%22min_regional_price%22:%7B%22field%22:%22data.gp_lowest_price_cents_2%22,%22aggregation%22:%22min%22%7D%7D%7D',
    headers=headers,
)

I don’t exactly like that it didn’t decode the URL parameters into Python dict, but this is a starting point for the script to be written.

We see that there’s a page parameter with values starting from one. This covers the API pagination aspect.

In case you are wondering about x-emb-id header it is related to mobile analytics solution provided by Embrace Mobile, Inc.

The next step is getting some product details via API. When we tap on one of the search results we see a bunch of HTTPS requests, most of them being made to subdomains of goat.com domain.

Screenshot 8

Let us explore what we have here. There’s a request to product template endpoint /api/v1/product_templates/.../show_v2 that takes product URL slug (e.g. terrex-swift-r2-mid-gtx-triple-black-cm7500) as part of URL path and gives the app a numeric product template ID to be used in some of the further API calls. It also provides some data is common across all product variants. To scrape it, our code will reproduce this API call with slug value it got from search API.

Like we did with search API request, we export this one to curl snippet:

curl -H 'x-px-authorization: 3' -H 'accept: application/json' -H 'authorization: Token token=""' --compressed -H 'accept-language: en-GB,en;q=0.9' -H 'x-emb-st: 1691934124434' -H 'user-agent: GOAT/2.62.0 (iPhone; iOS 16.6; Scale/2.00) Locale/en' -H 'x-emb-id: A131256965044D838D97E9AEC3CC32DE' -H 'x-px-original-token: 3:7b9f8feffc454bb265869bb69319201a10c0733ded5f64415904867ca6015448:V3kOFKugd0IYEzhYfgTK4QOh8dWCzZH04C4uoGYEfOekVmjMvCYLle7yVImUv8bSOoVChlY3FPELVmFZLboPxA==:1000:V6naWWAGfhIA54bPIFXyWPSpd7e9WmoWghqXoB1xwiAb0TVePEULt5nHoZFhWkpg1E4ZjMtwt1N9yfV2HCYOklHUqUy+oaAlYkACXQLwqsD21d70W55yb0UY9qHQHxY9zQcr6th//3ckUVLU/v1yWhZt/GV9jNyf6EesLG9fw+gqMWPhrpi8bDT1j5eeTR9BLmWMqrY3hmQSYRc9C7K5pQ==' -H 'cookie: __cf_bm=SHyG5WOKr777DofsXNbK3U29rw2zT0.FAfklx3N4xlA-1691934028-0-AUJcAbBlN7VoM9Sv8KU4ADRc7kRMROE5dN8u/rsVpl+cxdtwaLQTX/D/tIGlWKDEjaZfMbkE+PaMCzCrlGheIbI=' -H 'cookie: currency=EUR' https://www.goat.com/api/v1/product_templates/terrex-swift-r2-mid-gtx-triple-black-cm7500/show_v2

We also convert this snippet to Python:

import requests

cookies = {
    '__cf_bm': 'SHyG5WOKr777DofsXNbK3U29rw2zT0.FAfklx3N4xlA-1691934028-0-AUJcAbBlN7VoM9Sv8KU4ADRc7kRMROE5dN8u/rsVpl+cxdtwaLQTX/D/tIGlWKDEjaZfMbkE+PaMCzCrlGheIbI=',
    'currency': 'EUR',
}

headers = {
    'x-px-authorization': '3',
    'accept': 'application/json',
    'authorization': 'Token token=""',
    'accept-language': 'en-GB,en;q=0.9',
    'x-emb-st': '1691934124434',
    'user-agent': 'GOAT/2.62.0 (iPhone; iOS 16.6; Scale/2.00) Locale/en',
    'x-emb-id': 'A131256965044D838D97E9AEC3CC32DE',
    'x-px-original-token': '3:7b9f8feffc454bb265869bb69319201a10c0733ded5f64415904867ca6015448:V3kOFKugd0IYEzhYfgTK4QOh8dWCzZH04C4uoGYEfOekVmjMvCYLle7yVImUv8bSOoVChlY3FPELVmFZLboPxA==:1000:V6naWWAGfhIA54bPIFXyWPSpd7e9WmoWghqXoB1xwiAb0TVePEULt5nHoZFhWkpg1E4ZjMtwt1N9yfV2HCYOklHUqUy+oaAlYkACXQLwqsD21d70W55yb0UY9qHQHxY9zQcr6th//3ckUVLU/v1yWhZt/GV9jNyf6EesLG9fw+gqMWPhrpi8bDT1j5eeTR9BLmWMqrY3hmQSYRc9C7K5pQ==',
    # 'cookie': '__cf_bm=SHyG5WOKr777DofsXNbK3U29rw2zT0.FAfklx3N4xlA-1691934028-0-AUJcAbBlN7VoM9Sv8KU4ADRc7kRMROE5dN8u/rsVpl+cxdtwaLQTX/D/tIGlWKDEjaZfMbkE+PaMCzCrlGheIbI=; currency=EUR',
}

response = requests.get(
    'https://www.goat.com/api/v1/product_templates/terrex-swift-r2-mid-gtx-triple-black-cm7500/show_v2',
    cookies=cookies,
    headers=headers,
)

For the sake of example, we will save product name, main image URL and SKU from the API response here.

We also want to save some price data and availability data. The part of UI that shows prices and last sale amounts is populated by the /api/v2/product_variants/buy_bar_data endpoint. We extract the curl snippet from mitmproxy again:

curl -H 'x-px-authorization: 3' -H 'accept: application/json' -H 'authorization: Token token=""' --compressed -H 'accept-language: en-GB,en;q=0.9' -H 'x-emb-st: 1691934124855' -H 'user-agent: GOAT/2.62.0 (iPhone; iOS 16.6; Scale/2.00) Locale/en' -H 'x-emb-id: 9E8BF79CF66F4912A122C3C38F872E0E' -H 'x-px-original-token: 3:7b9f8feffc454bb265869bb69319201a10c0733ded5f64415904867ca6015448:V3kOFKugd0IYEzhYfgTK4QOh8dWCzZH04C4uoGYEfOekVmjMvCYLle7yVImUv8bSOoVChlY3FPELVmFZLboPxA==:1000:V6naWWAGfhIA54bPIFXyWPSpd7e9WmoWghqXoB1xwiAb0TVePEULt5nHoZFhWkpg1E4ZjMtwt1N9yfV2HCYOklHUqUy+oaAlYkACXQLwqsD21d70W55yb0UY9qHQHxY9zQcr6th//3ckUVLU/v1yWhZt/GV9jNyf6EesLG9fw+gqMWPhrpi8bDT1j5eeTR9BLmWMqrY3hmQSYRc9C7K5pQ==' -H 'cookie: __cf_bm=SHyG5WOKr777DofsXNbK3U29rw2zT0.FAfklx3N4xlA-1691934028-0-AUJcAbBlN7VoM9Sv8KU4ADRc7kRMROE5dN8u/rsVpl+cxdtwaLQTX/D/tIGlWKDEjaZfMbkE+PaMCzCrlGheIbI=' -H 'cookie: currency=EUR' 'https://www.goat.com/api/v1/product_variants/buy_bar_data?countryCode=US&productTemplateId=terrex-swift-r2-mid-gtx-triple-black-cm7500'

Screenshot 9 Screenshot 10

The corresponding Python snippet is:

import requests

cookies = {
    '__cf_bm': 'SHyG5WOKr777DofsXNbK3U29rw2zT0.FAfklx3N4xlA-1691934028-0-AUJcAbBlN7VoM9Sv8KU4ADRc7kRMROE5dN8u/rsVpl+cxdtwaLQTX/D/tIGlWKDEjaZfMbkE+PaMCzCrlGheIbI=',
    'currency': 'EUR',
}

headers = {
    'x-px-authorization': '3',
    'accept': 'application/json',
    'authorization': 'Token token=""',
    'accept-language': 'en-GB,en;q=0.9',
    'x-emb-st': '1691934124855',
    'user-agent': 'GOAT/2.62.0 (iPhone; iOS 16.6; Scale/2.00) Locale/en',
    'x-emb-id': '9E8BF79CF66F4912A122C3C38F872E0E',
    'x-px-original-token': '3:7b9f8feffc454bb265869bb69319201a10c0733ded5f64415904867ca6015448:V3kOFKugd0IYEzhYfgTK4QOh8dWCzZH04C4uoGYEfOekVmjMvCYLle7yVImUv8bSOoVChlY3FPELVmFZLboPxA==:1000:V6naWWAGfhIA54bPIFXyWPSpd7e9WmoWghqXoB1xwiAb0TVePEULt5nHoZFhWkpg1E4ZjMtwt1N9yfV2HCYOklHUqUy+oaAlYkACXQLwqsD21d70W55yb0UY9qHQHxY9zQcr6th//3ckUVLU/v1yWhZt/GV9jNyf6EesLG9fw+gqMWPhrpi8bDT1j5eeTR9BLmWMqrY3hmQSYRc9C7K5pQ==',
    # 'cookie': '__cf_bm=SHyG5WOKr777DofsXNbK3U29rw2zT0.FAfklx3N4xlA-1691934028-0-AUJcAbBlN7VoM9Sv8KU4ADRc7kRMROE5dN8u/rsVpl+cxdtwaLQTX/D/tIGlWKDEjaZfMbkE+PaMCzCrlGheIbI=; currency=EUR',
}

params = {
    'countryCode': 'LT',
    'productTemplateId': 'terrex-swift-r2-mid-gtx-triple-black-cm7500',
}

response = requests.get(
    'https://www.goat.com/api/v1/product_variants/buy_bar_data',
    params=params,
    cookies=cookies,
    headers=headers,
)

Quick experiment shows that removing __cf_bm cookie does not cause the requests to be blocked. However a default currency is set via another cookie, which we probably want to keep.

Since PerimeterX blocks requests based on source autonomous systems and IP reputation we need to route the requests through a proxy pool. I find that ISP proxies from Bright Data are good enough for this.

One last thing to note is that some APIs are meant to be used after login and thus don’t provide any data at this point.

A simple Python script to implement GOAT mobile API scraping is as follows:

#!/usr/bin/python3

import csv
import sys
from pprint import pprint
from urllib.parse import quote
import time

import requests

# ISP proxy URL
PROXY_URL = "[REDACTED]"

COUNTRY_CODE = "US"
CURRENCY = "USD"

FIELDNAMES = [
    "template_id",
    "slug",
    "name",
    "img_url",
    "brand",
    "sku",
    "box_condition",
    "shoe_condition",
    "size",
    "stock_status",
    "lowest_price",
]

RETRIES = 5

def create_session():
    session = requests.Session()

    session.headers = {
        "x-px-authorization": "3",
        "accept": "application/json",
        "authorization": 'Token token=""',
        "accept-language": "en-GB,en;q=0.9",
        "x-emb-st": "1691934124434",
        "user-agent": "GOAT/2.62.0 (iPhone; iOS 16.6; Scale/2.00) Locale/en",
        "x-emb-id": "A131256965044D838D97E9AEC3CC32DE",
        "x-px-original-token": "3:7b9f8feffc454bb265869bb69319201a10c0733ded5f64415904867ca6015448:V3kOFKugd0IYEzhYfgTK4QOh8dWCzZH04C4uoGYEfOekVmjMvCYLle7yVImUv8bSOoVChlY3FPELVmFZLboPxA==:1000:V6naWWAGfhIA54bPIFXyWPSpd7e9WmoWghqXoB1xwiAb0TVePEULt5nHoZFhWkpg1E4ZjMtwt1N9yfV2HCYOklHUqUy+oaAlYkACXQLwqsD21d70W55yb0UY9qHQHxY9zQcr6th//3ckUVLU/v1yWhZt/GV9jNyf6EesLG9fw+gqMWPhrpi8bDT1j5eeTR9BLmWMqrY3hmQSYRc9C7K5pQ==",
    }

    session.proxies = {"http": PROXY_URL, "https": PROXY_URL}

    session.cookies.set("currency", CURRENCY)

    return session


def get_product_template_data(session, product_slug):
    url = "https://www.goat.com/api/v1/product_templates/{}/show_v2".format(product_slug)

    resp = session.get(url)
    print(resp.url)

    return resp.json()

def get_buy_bar_data(session, product_slug):
    url = "https://www.goat.com/api/v1/product_variants/buy_bar_data"

    params = {
        'countryCode': COUNTRY_CODE,
        'productTemplateId': product_slug
    }

    tries_left = RETRIES

    while tries_left > 0:
        resp = session.get(url, params=params)
        print(resp.url)

        if resp.status_code == 200:
            break
        
        tries_left -= 1
        time.sleep(1)

    if resp.status_code == 200:
        return resp.json()
    else:
        return None

def scrape_search(session, search_query):
    page = 1

    url = "https://goat.cnstrc.com/search/" + quote(search_query)

    params = {
        "key": "key_XT7bjdbvjgECO5d8",
        "page": str(page),
        "fmt_options[hidden_fields]": "gp_instant_ship_lowest_price_cents_2",
        "fmt_options[hidden_facets]": "gp_instant_ship_lowest_price_cents_2",
        "features[display_variations]": "true",
        "feature_variants[display_variations]": "matched",
        "variations_map": '{"dtype":"object","group_by":[{"name":"product_condition","field":"data.product_condition"},{"name":"box_condition","field":"data.box_condition"}],"values":{"min_regional_instant_ship_price":{"field":"data.gp_instant_ship_lowest_price_cents_2","aggregation":"min"},"min_regional_price":{"field":"data.gp_lowest_price_cents_2","aggregation":"min"}}}',
    }
    
    while True:
        resp = session.get(url, params=params)
        print(resp.url)

        json_dict = resp.json()

        per_page = json_dict.get("request", dict()).get("num_results_per_page")

        results = json_dict.get("response", dict()).get("results", [])

        for result_dict in results:
            slug = result_dict.get("data", dict()).get("slug")
            
            yield slug

        if len(results) < per_page:
            break

        page += 1

        params['page'] = str(page)

def main():
    if len(sys.argv) != 2:
        print("Usage:")
        print("{} <query>".format(sys.argv[0]))
        return

    query = sys.argv[1]

    session = create_session()

    out_f = open("results.csv", "w", encoding="utf-8")

    csv_writer = csv.DictWriter(out_f, fieldnames=FIELDNAMES, lineterminator="\n")
    csv_writer.writeheader()

    for slug in scrape_search(session, query):
        print(slug)

        template_data = get_product_template_data(session, slug)

        template_id = template_data.get("id")
        name = template_data.get("name")
        img_url = template_data.get("mainPictureUrl")
        brand = template_data.get("sizeBrand")
        sku = template_data.get("sku")

        buy_bar_data = get_buy_bar_data(session, slug)

        for buy_bar_dict in buy_bar_data:
            box_condition = buy_bar_dict.get("boxCondition")
            shoe_condition = buy_bar_dict.get("shoeCondition")
            size = buy_bar_dict.get("sizeOption", dict()).get("value")
            stock_status = buy_bar_dict.get("stockStatus")
            lowest_price = buy_bar_dict.get("lowestPriceCents", dict()).get("amount")
            if type(lowest_price) == int:
                lowest_price = lowest_price / 100.0

            row = {
                "template_id": template_id,
                "slug": slug,
                "name": name,
                "img_url": img_url,
                "brand": brand,
                "sku": sku,
                "box_condition": box_condition,
                "shoe_condition": shoe_condition,
                "size": size,
                "stock_status": stock_status,
                "lowest_price": lowest_price
            }

            pprint(row)

            csv_writer.writerow(row)

    out_f.close()

if __name__ == "__main__":
    main()

Let us go through the code. The script gets the search query via CLI parameter (sys.argv[1]) and uses that to scrape URL slugs from the search API. This is done in scrape_search() function. The code here is based on the first Python snippet we got earlier. We implement page traversal by incrementing page parameter until the current page has less results that a full page would (threshold value is in request.num_results_per_page). This function is a generator function - by using yield instead of return we make it so that search API requests are done on as-needed basis and the function call can be iterated upon (which we do in main(), as technically the return value is an iterable generator object).

When we get the slug, scrape the two other endpoints for product template and pricing/availability/product condition data (a requests.Session object is used to keep the HTTP headers, cookies and proxy settings between requests). Since GOAT is a two-sided marketplace that allows reselling of used or defective shoes we also grab the fields about shoe and box condition. Data is merged into Python dict and written out into CSV file.

Screenshot 11

Now, there is PerimeterX SDK being integrated into GOAT mobile app. But as long we generate our traffic through good-enough proxies and reproduce the original headers we can scrape the data just fine, more or less. One problem is that sometimes 5 retries are not enough to get the data from buy_bar_data endpoint when PerimeterX starts asking for captcha solution. Modifying get_buy_bar_data() function to force proxy rotation on failed request is left as an exercise for the reader. But the larger point here is that anti-automation security measures on mobile app APIs are often not very tight and can be easily defeated even for prominent, well funded targets such as GOAT. To be fair it is harder to scrape behind login, but that’s a topic for another post.

Scrapy by example: scraping company data from Yelp

Sat, 12 Aug 2023 00:00:00 +0000

Yelp is a major yellow pages portal in USA and some other countries. It provides large amount of data on various businesses - phone numbers, addresses, descriptions, reviews, working hours and so on. Yelp provides a public API, but it is rate-limited to 500 requests per 24 hours. Thus we will be doing some web scraping to extract data from Yelp web pages.

Let us do some planning and strategy work first. This entails browsing the page with Chrome DevTools open, taking some notes and also loading some pages via Scrapy shell, e.g:

$ scrapy shell "https://www.yelp.com/search?find_desc=Pizza&find_loc=San+Francisco%2C+CA%2C+United+States"
2023-08-11 14:06:14 [scrapy.utils.log] INFO: Scrapy 2.8.0 started (bot: scrapybot)
2023-08-11 14:06:14 [scrapy.utils.log] INFO: Versions: lxml 4.9.2.0, libxml2 2.9.13, cssselect 1.2.0, parsel 1.7.0, w3lib 2.1.1, Twisted 22.10.0, Python 3.11.4 (main, Jun 20 2023, 17:23:00) [Clang 14.0.3 (clang-1403.0.22.14.1)], pyOpenSSL 23.1.1 (OpenSSL 3.1.0 14 Mar 2023), cryptography 40.0.1, Platform macOS-13.3.1-arm64-arm-64bit
2023-08-11 14:06:14 [scrapy.crawler] INFO: Overridden settings:
{'DUPEFILTER_CLASS': 'scrapy.dupefilters.BaseDupeFilter',
 'LOGSTATS_INTERVAL': 0}
2023-08-11 14:06:14 [py.warnings] WARNING: /opt/homebrew/lib/python3.11/site-packages/scrapy/utils/request.py:232: ScrapyDeprecationWarning: '2.6' is a deprecated value for the 'REQUEST_FINGERPRINTER_IMPLEMENTATION' setting.

It is also the default value. In other words, it is normal to get this warning if you have not defined a value for the 'REQUEST_FINGERPRINTER_IMPLEMENTATION' setting. This is so for backward compatibility reasons, but it will change in a future version of Scrapy.

See the documentation of the 'REQUEST_FINGERPRINTER_IMPLEMENTATION' setting for information on how to handle this deprecation.
  return cls(crawler)

2023-08-11 14:06:14 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.selectreactor.SelectReactor
2023-08-11 14:06:14 [scrapy.extensions.telnet] INFO: Telnet Password: 20481848a3d75456
2023-08-11 14:06:14 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.memusage.MemoryUsage']
2023-08-11 14:06:15 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2023-08-11 14:06:15 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2023-08-11 14:06:15 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2023-08-11 14:06:15 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
2023-08-11 14:06:15 [scrapy.core.engine] INFO: Spider opened
2023-08-11 14:06:17 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yelp.com/search?find_desc=Pizza&find_loc=San+Francisco%2C+CA%2C+United+States> (referer: None)
2023-08-11 14:06:17 [asyncio] DEBUG: Using selector: KqueueSelector
[s] Available Scrapy objects:
[s]   scrapy     scrapy module (contains scrapy.Request, scrapy.Selector, etc)
[s]   crawler    <scrapy.crawler.Crawler object at 0x107083e10>
[s]   item       {}
[s]   request    <GET https://www.yelp.com/search?find_desc=Pizza&find_loc=San+Francisco%2C+CA%2C+United+States>
[s]   response   <200 https://www.yelp.com/search?find_desc=Pizza&find_loc=San+Francisco%2C+CA%2C+United+States>
[s]   settings   <scrapy.settings.Settings object at 0x106213ed0>
[s]   spider     <DefaultSpider 'default' at 0x108044bd0>
[s] Useful shortcuts:
[s]   fetch(url[, redirect=True]) Fetch URL and update local objects (by default, redirects are followed)
[s]   fetch(req)                  Fetch a scrapy.Request and update local objects 
[s]   shelp()           Shell help (print this help)
[s]   view(response)    View response in a browser
2023-08-11 14:06:17 [asyncio] DEBUG: Using selector: KqueueSelector

Note the sucessfull request here - it means we’re not blocked by any security mechanism (yet).

Screenshot 1

Like many other websites in yellow pages world, Yelp has search feature with pagination and links to company detail pages. There’s also a map on he right hand side. If we wanted to get the map annotation coordinates we could extract them from JSON that is cooked into the page source. Yelp frontend is API-driven to a degree, but we can load the full HTML for each page. In this example, we will be running XPath queries on the pages downloaded by Scrapy.

For example, we can extract links to company detail pages like this:

In [1]: response.xpath('//a[starts-with(@href, "/biz/")]/@href').getall()
Out[1]: 
['/biz/golden-boy-pizza-san-francisco?osq=Pizza',
 '/biz/golden-boy-pizza-san-francisco?osq=Pizza',
 '/biz/golden-boy-pizza-san-francisco?hrid=QySEJtXN3DGb0KEVAmxF6w&osq=Pizza',
 '/biz/golden-boy-pizza-san-francisco?hrid=QySEJtXN3DGb0KEVAmxF6w&osq=Pizza',
 '/biz/golden-boy-pizza-san-francisco?hrid=QySEJtXN3DGb0KEVAmxF6w&osq=Pizza',
 '/biz/golden-boy-pizza-san-francisco?hrid=QySEJtXN3DGb0KEVAmxF6w&osq=Pizza',
 '/biz/PTFxtXS47ZVRCdZIrEWvGw?show_platform_modal=True',
 '/biz/tonys-pizza-napoletana-san-francisco?osq=Pizza',
 '/biz/tonys-pizza-napoletana-san-francisco?osq=Pizza',

...

The XPath query here is based on company page URL path always starting with /biz/. We see some duplication here, but that’s not going to be a problem as Scrapy will deduplicate the requests for us.

We also need a link to next page of search results. That can be extracted from “>” element that we can match by aria-label being set to Next. The XPath query here is also rather simple:

In [2]: response.xpath('//a[@aria-label="Next"]/@href').get()
Out[2]: 'https://www.yelp.com/search?find_desc=Pizza&find_loc=San%20Francisco%2C%20CA%2C%20United%20States&start=10'

Screenshot 2

Now we can explore the company details page. Let us also load it in Scrapy shell:

In [3]: fetch("https://www.yelp.com/biz/flour-water-pizza-san-francisco?osq=Pizza")
2023-08-11 14:24:22 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yelp.com/biz/flour-water-pizza-san-francisco?osq=Pizza> (referer: None)

Again, we get a fairly simple case of HTML being created on the server with not much done on the client side to render the data. Thus we can also run some simple XPath queries here to extract some information on the company:

In [4]: response.xpath("//h1/text()").get()
Out[4]: 'Flour + Water Pizza'

In [5]: response.xpath('//div[./p/a[text()="Get Directions"]]/p[last()]/text()').get()
Out[5]: '532 Columbus Ave San Francisco, CA 94133'

In [6]: response.xpath('//img[contains(@src, "maps.googleapis.com")]/@src').get()
Out[6]: 'https://maps.googleapis.com/maps/api/staticmap?size=315x150&sensor=false&client=gme-yelp&language=en&scale=1&zoom=15&center=37.800084%2C-122.409429&markers=scale%3A1%7Cicon%3Ahttps%3A%2F%2Fyelp-images.s3.amazonaws.com%2Fassets%2Fmap-markers%2Fannotation_32x43.png%7C37.800084%2C-122.409429&signature=IGF-J6T8eel638VQaoUG3pY1BtA='

Screenshot 3 Screenshot 4

The little map from Google Maps is not interactive. It’s a static image generated by Google Maps API. The URL usually (but not always) has a center parameter with company location coordinates. In some cases this little map shows the service coverage area. We will be extracting company coordinates from the image source URL when possible.

Screenshot 5

We run the following two commands to generate a Scrapy project with spider boilerplate:

$ scrapy startproject recon .
$ scrapy genspider yelp yelp.com

We edit items.py file to create an item class for data we scrape from Yelp company page:

# Define here the models for your scraped items
#
# See documentation in:
# https://docs.scrapy.org/en/latest/topics/items.html

import scrapy


class YelpItem(scrapy.Item):
    # define the fields for your item here like:
    yelp_biz_id = scrapy.Field()
    category = scrapy.Field()
    name = scrapy.Field()
    description = scrapy.Field()
    phone_number = scrapy.Field()
    website = scrapy.Field()
    address = scrapy.Field()
    latitude = scrapy.Field()
    longitude = scrapy.Field()
    gmaps_img_url = scrapy.Field()
    yelp_url = scrapy.Field()

We got the usual fields here - company name, address, phone number, address website URL and so on. We also include the URL to page we scrape to enable checks on the scraped data.

The Scrapy spider code in yelp.py file is as follows:

import scrapy

from urllib.parse import urlencode, urlparse, parse_qsl

from recon.items import YelpItem


class YelpSpider(scrapy.Spider):
    name = "yelp"
    allowed_domains = ["yelp.com"]
    start_urls = ["http://yelp.com/"]

    locations = None
    queries = None

    def __init__(self, locations="United States", queries="Pizza|Coffee"):
        super().__init__()

        self.locations = locations.split("|")
        self.queries = queries.split("|")

    def start_requests(self):
        for location in self.locations:
            for query in self.queries:
                url = "https://www.yelp.com/search"

                params = {"find_desc": query, "find_loc": location}

                url = url + "?" + urlencode(params)

                yield scrapy.Request(
                    url, callback=self.parse_company_list, meta={"category": query}
                )

    def parse_company_list(self, response):
        category = response.meta.get("category")

        company_links = response.xpath(
            '//a[starts-with(@href, "/biz/")]/@href'
        ).getall()
        company_links = list(map(lambda cl: cl.split("?")[0], company_links))
        company_links = list(set(company_links))

        for company_link in company_links:
            yield response.follow(
                company_link,
                callback=self.parse_company_details,
                meta={"category": category},
            )

        next_page_url = response.xpath('//a[@aria-label="Next"]/@href').get()
        if next_page_url is not None:
            yield scrapy.Request(
                next_page_url,
                callback=self.parse_company_list,
                meta={"category": category},
            )

    def parse_company_details(self, response):
        category = response.meta.get("category")

        item = YelpItem()

        item["category"] = category

        item["yelp_biz_id"] = response.xpath(
            '//meta[@name="yelp-biz-id"]/@content'
        ).get()

        item["name"] = response.xpath("//h1/text()").get()

        item["description"] = response.xpath(
            '//meta[@property="og:description"]/@content'
        ).get()
        if item["description"] is not None:
            item["description"] = item["description"].replace("\n", " ").strip()

        item["phone_number"] = response.xpath(
            '//div[./p[text()="Phone number"]]/p[last()]/text()'
        ).get()
        item["website"] = response.xpath(
            '//div[./p[text()="Business website"]]/p[last()]/a/@href'
        ).get()
        if item["website"] is not None and item["website"].startswith("/biz_redir"):
            o = urlparse(item["website"])
            params = parse_qsl(o.query)
            params = dict(params)
            item["website"] = params.get("url")

        item["address"] = response.xpath(
            '//div[./p/a[text()="Get Directions"]]/p[last()]/text()'
        ).get()

        gmaps_img_url = response.xpath(
            '//img[contains(@src, "maps.googleapis.com")]/@src'
        ).get()

        o = urlparse(gmaps_img_url)
        gmaps_params = parse_qsl(o.query)
        gmaps_params = dict(gmaps_params)

        self.logger.debug(gmaps_params)

        center = gmaps_params.get("center")
        if center is None:
            try:
                center = gmaps_params.get("markers").split("|")[-1]
            except:
                pass

        if center is not None:
            latitude, longitude = center.split(",")

            item["latitude"] = latitude
            item["longitude"] = longitude

        item["gmaps_img_url"] = gmaps_img_url
        item["yelp_url"] = response.url

        yield item

This spider is parametrised by location(s) and search queries that can be passed via CLI. Since both can include commas in their values we expect the user to use pipe symbol as separator. This aspect is accomplished in the constructor method (__init__). Note that Yelp limits the amount of search results per query-location pair, which makes search space sharding potentially applicable here.

In start_requests(), a standard Scrapy spider method, we iterate across both values of both parameters in nested loops to generate requests to initial search pages. We pass parse_company_list() as callback to request. When Scrapy fetches the page, it calls this method to parse it. At this callback, we do the following:

We extract company page links (relative URLs) and generate requests to get them. This time callback is another method - parse_company_details.
We extract the URL of next search page if there is one and generate a request to it. Since it will be the same kind of page with company links, the callback is the same.

We use the meta dictionary to pass the original search query along the request flow, as we want to preserve it in the scraped data item. Commonly the query is a business category. That’s why we name it as such.

In the parse_company_details method we apply XPath queries based on the page structure to extract the data we need. We already discussed a small complication with coordinates, but there’s also something we need to do about the website URL. To track clicks to the company website, Yelp links it through a redirect. Thus we use urlparse and parse_qsl functions from urllib.parse module to get a clean, direct URL. The code to extract rest of the fields is rather straightforward and trivially based on DOM structure. If the XPath queries look puzzling to you please read an earlier post on the very basics of XPath.

But there’s a little problem we have not addressed yet. Yelp does not have any agressive countermeasures deployed and it does not use any commercial solutions against scraping, but it does one annoying thing - temporary IP bans on excessive traffic. Thus we need some proxies here. I find that DC proxies are enough, but it’s best to force exit IP randomisation (e.g. rotating the proxy pool exit on every request). This is implemented in the following Scrapy middleware:

# Define here the models for your spider middleware
#
# See documentation in:
# https://docs.scrapy.org/en/latest/topics/spider-middleware.html

from scrapy import signals

from recon.settings import (
    BRIGHT_DATA_ENABLED,
    BRIGHT_DATA_ZONE_USERNAME,
    BRIGHT_DATA_ZONE_PASSWORD,
)

from w3lib.http import basic_auth_header

import random


class BrightDataDownloaderMiddleware:
    @classmethod
    def from_crawler(cls, crawler):
        return cls()

    def process_request(self, request, spider):
        if not BRIGHT_DATA_ENABLED:
            return None

        request.meta["proxy"] = "http://brd.superproxy.io:22225"

        username = BRIGHT_DATA_ZONE_USERNAME + "-session-" + str(random.random())

        request.headers["Proxy-Authorization"] = basic_auth_header(
            username, BRIGHT_DATA_ZONE_PASSWORD
        )

Since I use Bright Data, this entails a little vendor lock-in. This is updated version of middleware code from the previous post. Note that -session- string and a random number is appended to proxy zone username. This approach is based on sample code from Bright Data.

To integrate this middleware into the Scrapy project, we have to edit the settings.py file as follows:

BRIGHT_DATA_ENABLED = True
BRIGHT_DATA_ZONE_USERNAME = "[REDACTED]"
BRIGHT_DATA_ZONE_PASSWORD = "[REDACTED]"

DOWNLOADER_MIDDLEWARES = {"recon.middlewares.BrightDataDownloaderMiddleware": 500}

I also decided to raise CONCURRENT_REQUESTS and RETRY_TIMES to make the scraping more agressive:

# Configure maximum concurrent requests performed by Scrapy (default: 16)
CONCURRENT_REQUESTS = 32

RETRY_TIMES = 10

This covers the possibility of some exit IPs getting blocked by allowing for more retries to be made. It also speeds up the scraping by doing more requests concurrently.

Lastly, I disabled cookie handling, because we don’t need cookies to scrape this site:

# Disable cookies (enabled by default)
COOKIES_ENABLED = False

I could probably have left this on, but now I eliminate the possibility of cookies being used to cross-correlate requests to thwart or hinder the scraping.

So if we wanted to scrape data about coffee shops and pizza places in both San Francisco, CA and New York City we could run the Scrapy project like this:

$ scrapy runspider -a "locations=San Francisco, CA|New York, NY" \
  -a "queries=pizza|coffee" recon/spiders/yelp.py -O yelp.csv

Back to the terminal: the new era of CLI and TUI software

Wed, 09 Aug 2023 00:00:00 +0000

With desktop GUI apps becoming generally becoming increasingly bloated and sluggish over time (largely thanks to growing popularity of Electron.js) some technical users are turning back to sofware using the simpler and leaner user interfaces - command line and pseudo-GUIs being rendered in ASCII/Unicode (text user interfaces). CLI tools have the additional benefit of being relatively easy to be used via subprocceses from automation scripts. We will take a look at some modern software that is meant be used via terminal.

Carbonyl: 21st Century Lynx

Remember Lynx, a text-only web browser to render web pages in simplified, text only form? Funny thing is, Lynx is still being maintained to some degree, but lack of support for client-side JavaScript largely makes it obsolete for browsing the modern web, even if we consider the lack of image rendering to be a feature.

Carbonyl is a Chromium fork that is a modern take on TUI-based web browser idea. If you have Docker installed you can easily try it out:

$ docker run --rm -ti fathyb/carbonyl https://trickster.dev

Screenshot 1 Screenshot 2

It works quite well with simpler sites, but it can run into issues when you trip something up in anti-automation solutions. Another limitation is that you cannot watch videos with sound.

ag, ripgrep - the new grep(1)

You use grep(1) when you want to search something in source code or text file(s). The Silver Search (ag) is a tool similar to grep, but designed to search for substrings in source code and optimised for that end.

Ripgrep is another grep-like tool to run fast searches for text in a directory tree (with regex support), while also respecting your .gitignore file.

bottom, btop, glances - better kind of top(1)

In Linux/Unix systems top(1) is a TUI tool that renders a list of running processes with their resource consumption numbers - similar to Activity Monitor in macOS or Task Manager in Windows. Many tech people are aware of htop - an alternative tool with enhanced interactivity.

But we can go even further. Bottom (btm) is a Rust program that renders resource consumption graphs in Unicode pseudo-graphics with a process list.

Screenshot 3

btop and glances are another programs similar to Bottom.

Terminal Image Viewer (tiv)

But what if you want to take a quick glance at pictures without having a GUI environment? Perhaps you have some web scraping project that involves downloading images and it’s bit of pain to use scp(1) or sftp(1) to download them from the server in order to preview them. Terminal Image Viewer is a C++ program that relies on features of modern terminals for actually quite good image rendering.

Screenshot 4

Note however that this does not work on default Terminal app on macOS - you will need iTerm for this.

HTTPie - command line API client

HTTPie is a CLI tool designed specifically for launching requests against RESTful APIs. Unline curl, which is more general tool, HTTPie provides a simple CLI that directly maps into aspects of HTTP requests you want to generate:

$ http GET https://ifconfig.me/all.json
HTTP/1.1 200 OK
Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
Via: 1.1 google
access-control-allow-origin: *
content-length: 268
content-type: application/json; charset=utf-8
date: Tue, 08 Aug 2023 12:17:07 GMT
server: istio-envoy
strict-transport-security: max-age=2592000; includeSubDomains
x-envoy-upstream-service-time: 1

{
    "encoding": "gzip, deflate, br",
    "forwarded": "[REDACTED], 34.160.111.145,35.191.2.1",
    "ip_addr": "[REDACTED]",
    "method": "GET",
    "mime": "*/*",
    "port": 60392,
    "remote_host": "unavailable",
    "user_agent": "HTTPie/3.2.2",
    "via": "1.1 google"
}

We can see it also pretty-prints the JSON payload in the response.

jq - CLI tool and DSL for JSON processing

If pretty-printing the JSON is not enough, you can use jq - a CLI tool that provides an entire Domain Specific Language for JSON wrangling.

tshark - CLI equivalent of Wireshark

Wireshark is a prominent packet sniffer used by network engineers, security people and software developers. TShark is a relatively obscure sibling of Wireshark that is CLI-based. Think of it as a tcpdump(1), but better. It can be used to not only capture network traffic, but to run various filters and statistical techniques for traffic analysis purposes.

TermShark

But what if you want something like Wireshark GUI in your terminal environment? TermShark is a TUI wrapper for TShark that renders a user interface similar to that of Wireshark.

Screenshot 5

mitmproxy

When doing exploration of web and mobile apps for security research or automation one may want to intercept API traffic to see what conversations between the app and remote servers. mitmproxy is an interactive TUI program that works as HTTP(S) proxy server meant to capture and inspect the HTTP(S) requests. It’s a simpler alternative to Burp Suite GUI app that you can also customise with your own plugin code.

neovim

Neovim is a modernised and more extensible addition to the vi/VIM text editor lineage with features like Language Server Protocol support, Lua scripting interface, improved usability, RPC API, system clipboard integration, built-in terminal emulator and so on.

GitHub CLI tool

There’s git(1), and there’s GitHub. The former is CLI tool and the latter is web-based platform. To bridge the gap between the two there’s a Github CLI tool for doing tasks like working with tickets, managing pull requests and configuring the CI system. This can be helpful when doing devops work to automate various aspects of software development lifecycle.

mapscii

But what if you want to see some maps in your terminal? You can do that with mapscii - a Node.JS program that renders OpenStreetMap data in terminal environment with mouse support. If you don’t want to install it via NPM you can access a test instance via telnet protocol:

$ telnet mapscii.me

Screenshot 6

Visidata

In scraping and automation it is very common to deal with tabular data in formats such as CSV files, Excel spreadsheets, SQLite databases and so on. Visidata is TUI application for doing exactly that. It does a good subset of what Excel can do - data filtering, rendering some charts (for example, Shift-F quickly renders a histogram of values in the current column), lettting you edit cells and so on.

Screenshot 7 Screenshot 8

Since Visidata is developed in Python you can install it via PIP. It is also available on packet managers of some Linux systems.

Radare2

Radare2 is a programmable binary reverse engineering toolkit for low-level hackers. Besides the main TUI program it also ships r2pipe - a Python module for scripting binary analysis workflows (disassembly, binary code analysis, runtime debugging).

Rich and Textual

These are all nice examples, but what if you want to develop something like this yourself? Writing ncurses-based C programs is not the most productive way to spend time in 2023. When it comes to Python CLI/TUI programming, there are two inter-related projects of interest:

Rich - a library for enhanced text formatting.
Textual - a TUI programming framework, based on Rich (with CSS support).

To get a glimpse of what it can do, you can run:

$ python3 -m textual

Screenshot 9

Restringer: modular JavaScript deobfuscator

Sun, 30 Jul 2023 00:00:00 +0000

Restringer is a modular Javascript deobfuscation tool that attempts to autodetect and undo some common JS obfuscation techniques. This tool was developed by PerimeterX for malware analysis purposes. Besides CLI tool and a NPM module (for custom deobfuscator development), Restringer is also available as web app. We will be taking a look into the inner workings of this tool for educational purposes.

The underlying data model that Restringer operates on is not based on Babel flavour of Abstract Syntax Tree. Instead, it uses a library called flAST (also developed by PerimeterX) to parse the code into flattened form of AST, modify it and regenerate the code. AST flattening is a key concept in the Restringer project. Unlike in Babel, ASTs are not represented as tree structures with references between nodes. Instead, it is represented as an array of nodes that point to related nodes via references:

$ node
Welcome to Node.js v20.4.0.
Type ".help" for more information.
> const {generateFlatAST} = require('flast');
undefined
> const ast = generateFlatAST(`console.log(1 + 2 + 3);`);
undefined
> ast
[
  <ref *1> Node {
    type: 'Program',
    start: 0,
    end: 23,
    range: [ 0, 23 ],
    body: [ [Node] ],
    sourceType: 'module',
    comments: [],
    srcClosure: [Function (anonymous)],
    nodeId: 0,
    childNodes: [ [Node] ],
    parentNode: null,
    parentKey: '',
    lineage: [],
    isScopeBlock: true,
    allScopes: { '0': [GlobalScope] },
    scope: <ref *2> GlobalScope {
      type: 'global',
      set: Map(0) {},
      taints: Map(0) {},
      dynamic: true,
      block: [Circular *1],
      through: [Array],
      variables: [],
      references: [Array],
      variableScope: [Circular *2],
      functionExpressionScope: false,
      directCallToEvalScope: false,
      thisFound: false,
      __left: null,
      upper: null,
      isStrict: false,
      childScopes: [],
      __declaredVariables: [WeakMap],
      implicit: [Object]
    }
  },
  <ref *3> Node {
    type: 'ExpressionStatement',
    start: 0,
    end: 23,
    range: [ 0, 23 ],
    expression: Node {
      type: 'CallExpression',
      start: 0,
      end: 22,
      range: [Array],
      callee: [Node],
      arguments: [Array],
      optional: false,
      nodeId: 2,
      childNodes: [Array],
      parentNode: [Circular *3],
      parentKey: 'expression',
      lineage: [Array],
      scope: [GlobalScope]
    },
    ...
  <ref *12> Node {
    type: 'Literal',
    start: 20,
    end: 21,
    range: [ 20, 21 ],
    value: 3,
    raw: '3',
    nodeId: 10,
    childNodes: [],
    parentNode: <ref *8> Node {
      type: 'BinaryExpression',
      start: 12,
      end: 21,
      range: [Array],
      left: [Node],
      operator: '+',
      right: [Circular *12],
      nodeId: 6,
      childNodes: [Array],
      parentNode: [Node],
      parentKey: 'arguments',
      lineage: [Array],
      scope: [GlobalScope]
    },
    parentKey: 'right',
    lineage: [ 0, 1, 2, 6 ],
    scope: <ref *2> GlobalScope {
      type: 'global',
      set: Map(0) {},
      taints: Map(0) {},
      dynamic: true,
      block: [Node],
      through: [Array],
      variables: [],
      references: [Array],
      variableScope: [Circular *2],
      functionExpressionScope: false,
      directCallToEvalScope: false,
      thisFound: false,
      __left: null,
      upper: null,
      isStrict: false,
      childScopes: [],
      __declaredVariables: [WeakMap],
      implicit: [Object]
    }
  }
]

Furthermore, flAST data model tracks variable scopes to associate symbol declarations with their uses.

To let us modify the code in this flat AST form, flAST provides us with another abstraction - the Arborist object. There are two steps to making a code change with flAST - we first mark the nodes we want to replace or delete, then call applyChanges() method. The README file provides a simple example:

const {generateFlatAST, generateCode, Arborist} = require('flast');
const ast = generateFlatAST(`console.log('Hello' + ' ' + 'there!');`);
const replacements = {
  'Hello': 'General',
  'there!': 'Kenobi',
};
const arborist = new Arborist(ast);
// Mark all relevant nodes for replacement.
ast.filter(n => n.type === 'Literal' && replacements[n.value]).forEach(n => arborist.markNode(n, {
  type: 'Literal',
  value: replacements[n.value],
  raw: `'${replacements[n.value]}'`,
}));
const numberOfChangesMade = arborist.applyChanges();
console.log(generateCode(arborist.ast[0]));  // console.log('General' + ' ' + 'Kenobi');

Note the key conceptual difference in how the code modification is done here vs. in Babel-based code: instead of traversing the AST recursively and applying the visitor function we would need to use an filter() API that is based on a functional programming approach.

Another library that Restringer depends on is Obfuscation Detector - also from PerimeterX. This library applies pattern matching on flattened ASTs to detect several obfuscation techniques. For example, src/detectors/arrayReplacements.js file contains the following code to detect string array mapping:

const {
	arrayHasMinimumRequiredReferences,
	findArrayDeclarationCandidates,
} = require(__dirname + '/sharedDetectionMethods');

const obfuscationName = 'array_replacements';

/**
 * Array Replacements obfuscation type has the following characteristics:
 * - An array (A) is with many strings is defined.
 * - There are many member expression references where the object is array A.
 *
 * @param {ASTNode[]} flatTree
 * @return {string} The obfuscation name if detected; empty string otherwise.
 */
function detectArrayReplacements(flatTree) {
	const candidates = findArrayDeclarationCandidates(flatTree);

	for (const c of candidates) {
		const refs = c.id.references.map(n => n.parentNode);
		if (arrayHasMinimumRequiredReferences(refs, c.id.name, flatTree)) return obfuscationName;
	}
	return '';
}

module.exports = detectArrayReplacements;

detectArrayReplacements() is a detector function that relies on two other functions: findArrayDeclarationCandidates and arrayHasMinimumRequiredReferences from sharedDetectionMethods.js file. The former is looking for the array expressions that look like they are used to do string array mapping by applying some checks within predicate of a filter() call:

/**
 * @param {ASTNode[]} flatTree
 * @return {ASTNode[]} Candidates matching the target profile of an array with more than a few items, all literals.
 */
function findArrayDeclarationCandidates(flatTree) {
	return flatTree.filter(n =>
		n.type === 'VariableDeclarator' &&
		n?.init?.type === 'ArrayExpression' &&
		arrayHasMeaningfulContentLength(n.init.elements, flatTree) &&
		!n.init.elements.some(el => el.type !== 'Literal'));
}

Let us unpack the predicate here. First, it checks if node type is VariableDeclarator. Then, it checks if node points to another node of type ArrayExpression at it’s init property. Next, it calls arrayHasMeaningfulContentLength() function to check if array length is at least 2% of the entire flat AST. Lastly, it performs a check to reject arrays containing members that are not literals.

The other function, arrayHasMinimumRequiredReferences() checks for high enough reference count to array in question:

/**
 * @param {ASTNode[]} references
 * @param {string} targetArrayName
 * @param {ASTNode[]} flatTree
 * @return {boolean} true if the target array has at least the minimum required references; false otherwise.
 */
function arrayHasMinimumRequiredReferences(references, targetArrayName, flatTree) {
	return references.filter(n =>
		n.type === 'MemberExpression' &&
		n.object.name === targetArrayName).length / flatTree.length * 100 >= minMeaningfulPercentageOfReferences;
}

So if a candidate array has above 2% (const minMeaningfulPercentageOfReferences = 2; // 2%) of it’s members referenced via member expressions, the detector function consider it to be used for string array mapping.

At this point we are familiar with the building blocks for Restringer, so let us take a look into Restringer itself. The code base exposes a number of modules that can be used for custom deobfucation development. The very same modules are into src/restringer.js file that implements the CLI tool. There are two kind of modules in Restringer: unsafe ones that rely on code evaluation inside isolated Node.JS VM sandbox and safe ones that do not need that.

One simple module implements dead code removal in src/modules/safe/removeDeadNodes.js file:

const relevantParents = ['VariableDeclarator', 'AssignmentExpression', 'FunctionDeclaration', 'ClassDeclaration'];

/**
 * Remove nodes code which is only declared but never used.
 * NOTE: This is a dangerous operation which shouldn't run by default, invokations of the so-called dead code
 * may be dynamically built during execution. Handle with care.
 * @param {Arborist} arb
 * @param {Function} candidateFilter (optional) a filter to apply on the candidates list
 * @return {Arborist}
 */
function removeDeadNodes(arb, candidateFilter = () => true) {
	for (let i = 0; i < arb.ast.length; i++) {
		const n = arb.ast[i];
		if (n.type === 'Identifier' &&
		relevantParents.includes(n.parentNode.type) &&
		(!n?.declNode?.references?.length && !n?.references?.length) &&
		candidateFilter(n)) {
			const parent = n.parentNode;
			// Do not remove root nodes as they might be referenced in another script
			if (parent.parentNode.type === 'Program') continue;
			arb.markNode(parent?.parentNode?.type === 'ExpressionStatement' ? parent.parentNode : parent);
		}
	}
	return arb;
}

module.exports = removeDeadNodes;

This merely marks the AST nodes for removal. Applying the changes is left to the caller.

Each module is like a building block for deobfuscator development. However, PerimeterX also provides bigger constructs called processors, that rely on at least one module to simplify code from a specific obfuscation solution (e.g. self-defending code from Obfuscator.io). There is not many of them - Restringer should primarily be considered a framework to develop custom deobfuscators instead of ready-made general purpose deobfuscator itself.

In the Restringer class there’s a deobfuscate() method that pretty much runs everything:

	/**
	 * Entry point for this class.
	 * Determine obfuscation type and run the pre- and post- processors accordingly.
	 * Run the deobfuscation methods in a loop until nothing more is changed.
	 * Normalize script to make it more readable.
	 * @param {boolean} clean (optional) Remove dead nodes after deobfuscation. Defaults to false.
	 * @return {boolean} true if the script was modified during deobfuscation; false otherwise.
	 */
	deobfuscate(clean = false) {
		this.determineObfuscationType();
		this._runProcessors(this._preprocessors);
		this._loopSafeAndUnsafeDeobfuscationMethods();
		this._runProcessors(this._postprocessors);
		if (this.modified && this.normalize) this.script = normalizeScript(this.script);
		if (clean) this.script = runLoop(this.script, [removeDeadNodes]);
		return this.modified;
	}

The private method _loopSafeAndUnsafeDeobfuscationMethods() is of particular interest, as this is where deobfuscation modules are applied. The implementation of this method is as follows:

	/**
	 * Make all changes which don't involve eval first in order to avoid running eval on probelmatic values
	 * which can only be detected once part of the script is deobfuscated. Once all the safe changes are made,
	 * continue to the unsafe changes.
	 * Since the unsafe modification may be overreaching, run them only once and try the safe methods again.
	 */
	_loopSafeAndUnsafeDeobfuscationMethods() {
		let modified, script;
		do {
			this.modified = false;
			script = runLoop(this.script, this._safeDeobfuscationMethods());
			script = runLoop(script, this._unsafeDeobfuscationMethods(), 1);
			if (this.script !== script) {
				this.modified = true;
				this.script = script;
			}
			if (this.modified) modified = true;
		} while (this.modified); // Run this loop until the deobfuscation methods stop being effective.
		this.modified = modified;
	}

We see that deobfuscation methods are being applied until they no longer change the code. Safe methods are prioritised over unsafe ones. The runLoop() function in another file does the heavy lifting at flat AST level while also making sure that it does not get into infinite loop.

So how good is Restringer is in practice? Well, it is able to deobfuscate JSFuck output, but does not go much further without any customisations.

For example, it fails to completely cleanup the output of Obfuscator.io on the “Low” preset:

$ restringer obfuscated.js 
[+] Obfuscation type is Generic
	[+] resolveProxyReferences committed 1 new changes!
	[+] resolveFunctionConstructorCalls committed 1 new changes!
[+] ==> Cycle 1 completed in 0.023 seconds with 2 changes (241 nodes)
[+] ==> Cycle 2 completed in 0.001 seconds with 2 changes (241 nodes)
	[+] resolveMinimalAlphabet committed 6 new changes!
[+] ==> Cycle 3 completed in 5.058 seconds with 6 changes (233 nodes)
	[+] resolveFunctionConstructorCalls committed 2 new changes!
[+] ==> Cycle 4 completed in 0.006 seconds with 2 changes (224 nodes)
[+] ==> Cycle 5 completed in 0 seconds with 2 changes (224 nodes)
	[+] resolveMinimalAlphabet committed 1 new changes!
[+] ==> Cycle 6 completed in 0.026 seconds with 1 changes (223 nodes)
[+] ==> Cycle 7 completed in 0.001 seconds with no changes (223 nodes)
[+] ==> Cycle 8 completed in 0.001 seconds with no changes (223 nodes)
	[+] normalizeComputed committed 12 new changes!
[+] ==> Cycle 9 completed in 0.007 seconds with 12 changes (223 nodes)
[+] ==> Cycle 10 completed in 0 seconds with 12 changes (223 nodes)
[+] Saved obfuscated.js-deob.js
[!] Deobfuscation took 5.184 seconds.
const _0x5d3fc2 = function () {
  let _0x5e1813 = true;
  return function (_0x4b42c2, _0x1f2986) {
    const _0xab51ad = _0x5e1813 ? function () {
      if (_0x1f2986) {
        const _0x18ac9b = _0x1f2986.apply(_0x4b42c2, arguments);
        _0x1f2986 = null;
        return _0x18ac9b;
      }
    } : function () {
    };
    _0x5e1813 = false;
    return _0xab51ad;
  };
}();
(function () {
  _0x5d3fc2(this, function () {
    const _0x1c8f0b = new RegExp('function *\\( *\\)');
    const _0x6cbb3 = new RegExp('\\+\\+ *(?:[a-zA-Z_$][0-9a-zA-Z_$]*)', 'i');
    const _0x496fd6 = _0x4aeb94('init');
    if (!_0x1c8f0b.test(_0x496fd6 + 'chain') || !_0x6cbb3.test(_0x496fd6 + 'input')) {
      _0x496fd6('0');
    } else {
      _0x4aeb94();
    }
  })();
}());
const verseChoose = document.querySelector('select');
const poemDisplay = document.querySelector('pre');
verseChoose.addEventListener('change', () => {
  const _0x4f5b1d = verseChoose.value;
  updateDisplay(verseChoose.value);
});
function _0x4aeb94(_0x662129) {
  function _0x9c1238(_0x2dfb67) {
    if (typeof _0x2dfb67 === 'string') {
      return function () {
        while (true) {
        }
      }.apply('counter');
    } else {
      if (('' + _0x2dfb67 / _0x2dfb67).length !== 1 || _0x2dfb67 % 20 === 0) {
        (function () {
          debugge_;
        }.call('action'));
      } else {
        (function () {
          debugge_;
        }.apply('stateObject'));
      }
    }
    _0x9c1238(++_0x2dfb67);
  }
  try {
    if (_0x662129) {
      return _0x9c1238;
    } else {
      _0x9c1238(0);
    }
  } catch (_0x45a2e1) {
  }
}

Compare and contrast the output here with the original snippet before it was obfuscated:

// https://developer.mozilla.org/en-US/docs/Learn/JavaScript/Client-side_web_APIs/Fetching_data
const verseChoose = document.querySelector("select");
const poemDisplay = document.querySelector("pre");

verseChoose.addEventListener("change", () => {
  const verse = verseChoose.value;
  updateDisplay(verse);
});

In this case Restringer really did not do much to undo what Obfuscator.io did.

This can perhaps be explained by Restringer being a rather specialised piece of software that is primarily developed for malware analysis. Furthermore, undoing code obfuscations is a problem of high inherent complexity. There’s so many ways one can make the code confusing to read and an ideal deobfuscator would need to address all of them, which is a practical impossibility.

However in my opinion using flat ASTs is interesting idea that deserves some attention. When developing software, it is of great importance to work out how exactly the underlying data structures will be organised, as it will determine what kind of algorithms will be applicable to process the data. That will have its own ramifications on how hard will it be to write the code and how fast it will run. In this particular case, choosing flat ASTs enabled using elements of functional programming that would not be applicable for regular ASTs that Babel parses the code into.

Using Scrapy-playwright for integrating browser automation with Scrapy

Sat, 22 Jul 2023 00:00:00 +0000

Scrapy framework provides a great deal of machinery for developing and operating web scrapers that is based on launching requests and parsing responses. However, sometimes it is desirable to introduce browser automation into a web scraping project. One may want to have code as general as possible across many target sites, address certain kinds of blocking (e.g. Javascript challenges) or simply take screenshots of pages being scraped. Scrapy-playwright is Scrapy plugin that connects Scrapy with MS Playwright browser automation software. In this post we will get familiar with it and go through some code examples that demonstrate the integration between the two.

The following steps require Scrapy, Playwright and Scrapy-playwright to be installed in your system.

To enable Scrapy-playwright in the Scrapy project, we have to override the DOWNLOADER_HANDLERS configuration parameter to the following:

DOWNLOAD_HANDLERS = {
    "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
    "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
}

Furthermore, we must also override TWISTED_REACTOR:

TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"

This replaces a class for reactor object class that is used by Scrapy. Reactor object is where network event loop is running for the purposes of Twisted network programming framework that Scrapy is based upon.

Now we can use Playwright integration for launching some or all of the requests through the browser. If we want the request to be performed through browser, we add "playwright": True to the meta dictionary for that request.

Let us go through a simple example of Scrapy-based script that relies on Playwright to achieve greater scraper generality when checking for presence of user-provided keyword on the pages.

Take a look at this code:

#!/usr/bin/python3

import argparse
from datetime import datetime
import sys
import os
from urllib.parse import urlparse
import logging

import scrapy
from scrapy.crawler import CrawlerProcess


class KeywordSpider(scrapy.Spider):
    name = "kwchecker"
    start_urls = []
    allowed_domains = []
    keywords = []
    use_playwright = False

    def __init__(self, start_urls, keywords, use_playwright, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.start_urls = list(set(start_urls))

        allowed_domains = []
        for start_url in self.start_urls:
            domain = urlparse(start_url).netloc
            allowed_domains.append(domain)

        self.allowed_domains = list(set(allowed_domains))
        self.keywords = list(set(keywords))
        self.use_playwright = use_playwright

    def start_requests(self):
        meta = {"playwright": self.use_playwright}
        for start_url in self.start_urls:
            yield scrapy.Request(start_url, meta=meta)

    def parse(self, response):
        meta = dict()
        if response.meta.get("playwright") is not None:
            meta["playwright"] = response.meta.get("playwright")
        try:
            text = " ".join(response.xpath("*//text()").getall()).strip()
        except:
            return

        for keyword in self.keywords:
            if keyword.lower() in text.lower():
                yield {
                    "keyword": keyword,
                    "url": response.url,
                    "seen_at": datetime.now().isoformat(),
                }

        links = response.xpath("//a/@href").getall()
        for link in links:
            o = urlparse(link)
            if o.scheme == "" or o.scheme == "http" or o.scheme == "https":
                yield response.follow(link, meta=meta)


def read_lines(file_path):
    in_f = open(file_path, "r")
    lines = in_f.read().strip().split("\n")
    in_f.close()

    lines = list(map(lambda line: line.strip(), lines))
    lines = list(set(lines))

    return lines


def main():
    os.chdir(os.path.dirname(os.path.realpath(__file__)))

    parser = argparse.ArgumentParser()

    parser.add_argument(
        "--start-urls",
        required=False,
        help="List of start URLs, one per line",
        default="start_urls.txt",
    )
    parser.add_argument(
        "--keywords",
        required=False,
        help="List of keywords, one per line",
        default="keywords.txt",
    )
    parser.add_argument(
        "--use-playwright",
        required=False,
        help="Use Playwright",
        default=False,
        action="store_true",
    )
    parser.add_argument(
        "--output-csv-file",
        required=False,
        help="Output CSV file",
        default="output.csv",
    )

    parsed_args = vars(parser.parse_args(sys.argv[1:]))
    print("Parsed CLI args: {}".format(parsed_args))

    start_urls = read_lines(parsed_args.get("start_urls"))
    keywords = read_lines(parsed_args.get("keywords"))
    output_csv_file = parsed_args.get("output_csv_file")
    use_playwright = parsed_args.get("use_playwright")

    if os.path.isfile(output_csv_file):
        os.unlink(output_csv_file)

    headers = {
        "accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7",
        "accept-language": "en-GB,en-US;q=0.9,en;q=0.8",
        "cache-control": "no-cache",
        "pragma": "no-cache",
        "sec-ch-ua": '"Google Chrome";v="111", "Not(A:Brand";v="8", "Chromium";v="111"',
        "sec-ch-ua-mobile": "?0",
        "sec-ch-ua-platform": '"macOS"',
        "sec-fetch-dest": "document",
        "sec-fetch-mode": "navigate",
        "sec-fetch-site": "none",
        "sec-fetch-user": "?1",
        "upgrade-insecure-requests": "1",
        "user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36",
    }

    settings = {
        "USER_AGENT": headers.get("user-agent"),
        "DEFAULT_REQUEST_HEADERS": headers,
        "FEEDS": {output_csv_file: {"format": "csv"}},
        "REQUEST_FINGERPRINTER_IMPLEMENTATION": "2.7",
    }

    if use_playwright:
        settings["DOWNLOAD_HANDLERS"] = {
            "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
            "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
        }

        settings[
            "TWISTED_REACTOR"
        ] = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"

    process = CrawlerProcess(settings=settings)

    process.crawl(KeywordSpider, start_urls, keywords, use_playwright)
    process.start()


if __name__ == "__main__":
    main()

For the sake of simplicity, we use CrawlerProcess to run a single Scrapy spider for a script instead of creating a Scrapy project that has far more boilerplate. We override the aforementioned settings, as well as USER_AGENT and DEFAULT_REQUEST_HEADERS to make our scraper look more like a normal browser. Note that the user of this script is allowed to choose not to use browser automation by not passing in --use-playwright CLI option. Scrapy spider works pretty much the same, except that value for playwright key in meta dictionary is set to False.

When running this script we can see some output that is a little different than what we regulary see from Scrapy:

$ kw_checker.py --use-playwright --start-urls start_urls.txt --keywords keywords.txt --output-csv-file results.csv
Parsed CLI args: {'start_urls': 'start_urls.txt', 'keywords': 'keywords.txt', 'use_playwright': True, 'output_csv_file': 'results.csv'}
2023-07-21 16:26:09 [scrapy.utils.log] INFO: Scrapy 2.8.0 started (bot: scrapybot)
2023-07-21 16:26:09 [scrapy.utils.log] INFO: Versions: lxml 4.9.2.0, libxml2 2.9.13, cssselect 1.2.0, parsel 1.7.0, w3lib 2.1.1, Twisted 22.10.0, Python 3.11.4 (main, Jun 20 2023, 17:23:00) [Clang 14.0.3 (clang-1403.0.22.14.1)], pyOpenSSL 23.1.1 (OpenSSL 3.1.0 14 Mar 2023), cryptography 40.0.1, Platform macOS-13.3.1-arm64-arm-64bit
2023-07-21 16:26:09 [scrapy.crawler] INFO: Overridden settings:
{'REQUEST_FINGERPRINTER_IMPLEMENTATION': '2.7',
 'TWISTED_REACTOR': 'twisted.internet.asyncioreactor.AsyncioSelectorReactor',
 'USER_AGENT': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) '
               'AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 '
               'Safari/537.36'}
2023-07-21 16:26:09 [asyncio] DEBUG: Using selector: KqueueSelector
2023-07-21 16:26:09 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.asyncioreactor.AsyncioSelectorReactor
2023-07-21 16:26:09 [scrapy.utils.log] DEBUG: Using asyncio event loop: asyncio.unix_events._UnixSelectorEventLoop
2023-07-21 16:26:09 [scrapy.extensions.telnet] INFO: Telnet Password: 8ed078a7779ff379
2023-07-21 16:26:09 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.memusage.MemoryUsage',
 'scrapy.extensions.feedexport.FeedExporter',
 'scrapy.extensions.logstats.LogStats']
2023-07-21 16:26:10 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2023-07-21 16:26:10 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2023-07-21 16:26:10 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2023-07-21 16:26:10 [scrapy.core.engine] INFO: Spider opened
2023-07-21 16:26:10 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2023-07-21 16:26:10 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
2023-07-21 16:26:10 [scrapy-playwright] INFO: Starting download handler
2023-07-21 16:26:10 [scrapy-playwright] INFO: Starting download handler
2023-07-21 16:26:15 [scrapy-playwright] INFO: Launching browser chromium
2023-07-21 16:26:15 [scrapy-playwright] INFO: Browser chromium launched
2023-07-21 16:26:15 [scrapy-playwright] DEBUG: Browser context started: 'default' (persistent=False)
2023-07-21 16:26:15 [scrapy-playwright] DEBUG: [Context=default] New page created, page count is 1 (1 for all contexts)
2023-07-21 16:26:15 [scrapy-playwright] DEBUG: [Context=default] New page created, page count is 2 (2 for all contexts)
2023-07-21 16:26:15 [scrapy-playwright] DEBUG: [Context=default] Request: <GET http://quotes.toscrape.com/js/> (resource type: document, referrer: None)
2023-07-21 16:26:15 [scrapy-playwright] DEBUG: [Context=default] Request: <GET http://books.toscrape.com/> (resource type: document, referrer: None)
2023-07-21 16:26:16 [scrapy-playwright] DEBUG: [Context=default] Response: <200 http://books.toscrape.com/> (referrer: None)
2023-07-21 16:26:16 [scrapy-playwright] DEBUG: [Context=default] Response: <200 http://quotes.toscrape.com/js/> (referrer: None)
2023-07-21 16:26:16 [scrapy-playwright] DEBUG: [Context=default] Request: <GET http://books.toscrape.com/static/oscar/css/styles.css> (resource type: stylesheet, referrer: http://books.toscrape.com/)
2023-07-21 16:26:16 [scrapy-playwright] DEBUG: [Context=default] Request: <GET http://books.toscrape.com/static/oscar/js/bootstrap-datetimepicker/bootstrap-datetimepicker.css> (resource type: stylesheet, referrer: http://books.toscrape.com/)
2023-07-21 16:26:16 [scrapy-playwright] DEBUG: [Context=default] Request: <GET http://books.toscrape.com/static/oscar/css/datetimepicker.css> (resource type: stylesheet, referrer: http://books.toscrape.com/)
2023-07-21 16:26:16 [scrapy-playwright] DEBUG: [Context=default] Request: <GET http://quotes.toscrape.com/static/bootstrap.min.css> (resource type: stylesheet, referrer: http://quotes.toscrape.com/js/)
2023-07-21 16:26:16 [scrapy-playwright] DEBUG: [Context=default] Request: <GET http://quotes.toscrape.com/static/main.css> (resource type: stylesheet, referrer: http://quotes.toscrape.com/js/)
2023-07-21 16:26:16 [scrapy-playwright] DEBUG: [Context=default] Request: <GET http://quotes.toscrape.com/static/jquery.js> (resource type: script, referrer: http://quotes.toscrape.com/js/)

We see that a headless Chromium browser is launched and that a browser context is being mentioned. A browser context is essentially a session within a browser that is controlled by Playwright.

As we would expect, images and JavaScript files are downloaded by the browser without spider explicitly asking for that:

2023-07-21 16:32:53 [scrapy-playwright] DEBUG: [Context=default] Response: <200 http://books.toscrape.com/static/oscar/js/bootstrap-datetimepicker/locales/bootstrap-datetimepicker.all.js> (referrer: None)
2023-07-21 16:32:53 [scrapy-playwright] DEBUG: [Context=default] Response: <200 http://books.toscrape.com/media/cache/ee/a9/eea9e831f8964b4dc0190c84a1f9a1f6.jpg> (referrer: None)
2023-07-21 16:32:53 [scrapy-playwright] DEBUG: [Context=default] Response: <200 http://books.toscrape.com/media/cache/dc/4d/dc4d070e33813a07a4e02f069e6d482f.jpg> (referrer: None)

Self-defending JS code and debugger traps

Sun, 02 Jul 2023 00:00:00 +0000

Sometimes programmers really don’t want their code to tampered with during runtime. Code obfuscation makes it harder to read and understand, but by itself does not do anything against modification and dynamic analysis with a debugger. This is particularly applicable to client-side JavaScript code that is meant to be running in web browser environment. After all, the code is largely under users control after it is fetched from the server. We will look into two kinds of techniques to make this kind of tampering harder: self-defending code to hinder code modifications and debugger traps are meant to mess with debugger if someone tries to dynamically step through the code during runtime. Whitehat developers at security or anti-fraud software companies would be interested in these techniques for the same reason as the blackhat malware writers: to hinder their adversaries from discovering the inner working of the code and developing attacks or defenses against it.

We can use obfuscator.io to get simple examples of both techniques. Let us start with the following very basic unprotected snippet:

// Paste your JavaScript code here
function hi() {
  console.log("Hello World!");
  console.log("Hello World!");
  console.log("Hello World!");
  console.log("Hello World!");
  console.log("Hello World!");
  console.log("Hello World!");
  console.log("Hello World!");
}
hi();

On the obfuscator web UI we uncheck all checkboxes except “Self Defending” and “Compact”. We get the following obfuscated snippet:

function hi(){var _0x3298fd=(function(){var _0x3c6e04=!![];return function(_0x471481,_0x2bd7a3){var _0x14b95c=_0x3c6e04?function(){if(_0x2bd7a3){var _0xc92697=_0x2bd7a3['apply'](_0x471481,arguments);_0x2bd7a3=null;return _0xc92697;}}:function(){};_0x3c6e04=![];return _0x14b95c;};}());var _0x3a8b12=_0x3298fd(this,function(){return _0x3a8b12['toString']()['search']('(((.+)+)+)+$')['toString']()['constructor'](_0x3a8b12)['search']('(((.+)+)+)+$');});_0x3a8b12();console['log']('Hello\x20World!');console['log']('Hello\x20World!');console['log']('Hello\x20World!');console['log']('Hello\x20World!');console['log']('Hello\x20World!');console['log']('Hello\x20World!');console['log']('Hello\x20World!');}hi();

Screenshot 1

Undoing the minification makes it bit cleaner:

function hi() {
  var _0xd9587a = (function() {
    var _0x505420 = !![];
    return function(_0x1ae0dc, _0x3c054) {
      var _0x1c2bf6 = _0x505420 ? function() {
        if (_0x3c054) {
          var _0x6b34ef = _0x3c054['apply'](_0x1ae0dc, arguments);
          _0x3c054 = null;
          return _0x6b34ef;
        }
      } : function() {};
      _0x505420 = ![];
      return _0x1c2bf6;
    };
  }());
  var _0x23cc70 = _0xd9587a(this, function() {
    return _0x23cc70['toString']()['search']('(((.+)+)+)+$')['toString']()['constructor'](_0x23cc70)['search']('(((.+)+)+)+$');
  });
  _0x23cc70();
  console['log']('Hello\x20World!');
  console['log']('Hello\x20World!');
  console['log']('Hello\x20World!');
  console['log']('Hello\x20World!');
  console['log']('Hello\x20World!');
  console['log']('Hello\x20World!');
  console['log']('Hello\x20World!');
}
hi();

We see some weird obfuscated code injected in here. Let us explore it a bit with Node.JS debugger:

$ node inspect self_defending_edited.js
< Debugger listening on ws://127.0.0.1:9229/d6b71c95-1217-41e4-b188-e30a7a822b47
< For help, see: https://nodejs.org/en/docs/inspector
< 
< Debugger attached.
< 
 ok
Break on start in self_defending_edited.js:28
 26   console['log']('Hello\x20World!');
 27 }
>28 hi();
 29 
debug> s
break in self_defending_edited.js:2
  1 function hi() {
> 2   var _0xd9587a = (function() {
  3     var _0x505420 = !![];
  4     return function(_0x1ae0dc, _0x3c054) {
debug> s

...

debug> s
break in self_defending_edited.js:17
 15   }());
 16   var _0x23cc70 = _0xd9587a(this, function() {
>17     return _0x23cc70['toString']()['search']('(((.+)+)+)+$')['toString']()['constructor'](_0x23cc70)['search']('(((.+)+)+)+$');
 18   });
 19   _0x23cc70();
debug> exec _0x23cc70
[Function: function]
debug> exec _0x23cc70.toString()
'function() {\n' +
  '        if (_0x3c054) {\n' +
  "          var _0x6b34ef = _0x3c054['apply'](_0x1ae0dc, arguments);\n" +
  '          _0x3c054 = null;\n' +
  '          return _0x6b34ef;\n' +
  '        }\n' +
  '      }'
debug> s
debug> s
Uncaught Error [ERR_DEBUGGER_ERROR]: Can only perform operation while paused.
    at _pending.<computed> (node:internal/debugger/inspect_client:247:27)
    at Client._handleChunk (node:internal/debugger/inspect_client:214:11)
    at Socket.emit (node:events:511:28)
    at Socket.emit (node:domain:489:12)
    at addChunk (node:internal/streams/readable:332:12)
    at readableAddChunk (node:internal/streams/readable:305:9)
    at Readable.push (node:internal/streams/readable:242:10)
    at TCP.onStreamRead (node:internal/stream_base_commons:190:23)
    at TCP.callbackTrampoline (node:internal/async_hooks:130:17) {
  code: -32000
}

We see that it crashed when attempting to match regular expression (((.+)+)+)+$ on source code string of one of the nested functions. Since we undid the minification and introduced white space characters into the code, this operation overwhelms the regular expression engine by causing catastrophic backtracking.

More generally, self-defending code involves self-introspection, integrity checking and triggering anti-tampering countermeasures. In this case the latter two were done together in the weird regex statement, but they can be separate. Integrity checking may involve hashing key parts of the code and checking against a precomputed hash value. Anti-tampering countermeasures may involve throwing an expection to crash the code or redirecting the user to error page.

To get an example of a debugger trap, we rerun the obfuscation with only a “Debug Protection” feature being activated.

Screenshot 2

We get the following snippet:

function hi() {
    var _0x20b38b = (function () {
        var _0x36b5cf = !![];
        return function (_0x56f748, _0x5b111c) {
            var _0x37844a = _0x36b5cf ? function () {
                if (_0x5b111c) {
                    var _0x4a1f4c = _0x5b111c['apply'](_0x56f748, arguments);
                    _0x5b111c = null;
                    return _0x4a1f4c;
                }
            } : function () {
            };
            _0x36b5cf = ![];
            return _0x37844a;
        };
    }());
    (function () {
        _0x20b38b(this, function () {
            var _0x4ac2e9 = new RegExp('function\x20*\x5c(\x20*\x5c)');
            var _0x3b4dbb = new RegExp('\x5c+\x5c+\x20*(?:[a-zA-Z_$][0-9a-zA-Z_$]*)', 'i');
            var _0x43549a = _0x4c804e('init');
            if (!_0x4ac2e9['test'](_0x43549a + 'chain') || !_0x3b4dbb['test'](_0x43549a + 'input')) {
                _0x43549a('0');
            } else {
                _0x4c804e();
            }
        })();
    }());
    console['log']('Hello\x20World!');
    console['log']('Hello\x20World!');
    console['log']('Hello\x20World!');
    console['log']('Hello\x20World!');
    console['log']('Hello\x20World!');
    console['log']('Hello\x20World!');
    console['log']('Hello\x20World!');
}
hi();
function _0x4c804e(_0x203e97) {
    function _0x21f924(_0x6c4ed) {
        if (typeof _0x6c4ed === 'string') {
            return function (_0x194661) {
            }['constructor']('while\x20(true)\x20{}')['apply']('counter');
        } else {
            if (('' + _0x6c4ed / _0x6c4ed)['length'] !== 0x1 || _0x6c4ed % 0x14 === 0x0) {
                (function () {
                    return !![];
                }['constructor']('debu' + 'gger')['call']('action'));
            } else {
                (function () {
                    return ![];
                }['constructor']('debu' + 'gger')['apply']('stateObject'));
            }
        }
        _0x21f924(++_0x6c4ed);
    }
    try {
        if (_0x203e97) {
            return _0x21f924;
        } else {
            _0x21f924(0x0);
        }
    } catch (_0x5a4b0c) {
    }
}

The injected code is somewhat obfuscated, but we have a small snippet here and can get an idea of what happens here. The obfuscator injected a code that performs a classical anti-debugging trick - invoking debugger JS statement periodically. This will disrupt an attempt to step through the code one line at a time like in web browser dev tools panel and may even crash the entire browser on a debugging attempt.

None of the above techniques are silver bullets against reverse engineer looking to work out what exactly a malware sample does or how to defeat security mechanisms of automation-hostile website. They merely make it incrementally harder and can be dealt with by removing the injected code at AST level or even by manually editing the source.

Abusing Ahrefs Backlink Checker

Tue, 27 Jun 2023 00:00:00 +0000

World Wide Web is a network of HTML documents (pages) with hyperlinks between them. Consider a directed graph that consists of vertices representing pages and edges representing links between pages. For a given page, links from other pages to that page are known as backlinks. Backlinks are of significance to search engine ranking of the site, thus making them of importance to SEO people, digital marketers and growth hackers. Furthermore, OSINT practitioners and web scraper developers may want to find sites/pages linking to certain pages/files on the web as it would help traversing the data landscape. Major SEO SaaS apps (Ahrefs, SEMRush, Moz, etc.) are sourcing their data via large-scale web crawling operations that build up a fairly complete maps of the web and thus can be leveraged for this purpose. In this post we will go through an example of abusing a free Ahrefs Backlink Checker to show how a list of backlinks can be scraped from a SEO tool.

Let us explore the request flow that happens when the user pastes an URL into the form and presses the “Check backlinks” button. We are dealing with Cloudflare Turnstile - a drop-in alternative to Google reCaptcha that may ask for checkbox to be clicked, but largely tries to separate bots from humans by non-interactive means - probing the JS environment in the browser, JavaScript challenges, gathering a sample of user activity and so on. Since it is kind of captcha, we can use Anti-captcha service to solve it. For $2 we can buy a thousand solutions. In Chrome DevTools we can see a GET request to:

https://challenges.cloudflare.com/cdn-cgi/challenge-platform/h/g/turnstile/if/ov2/av0/rcv0/0/dy268/0x4AAAAAAAAzi9ITzSN9xKMi/light/normal

Value 0x4AAAAAAAAzi9ITzSN9xKMi is a site key - a unique value that identifies this site in API interactions with Cloudflare. We will be passing it into the Anti-captcha API.

Screenshot 1

Once the site gets a captcha token from Cloudflare it launches two HTTP POST requests to the API. We extract both of them as curl snippets to be converted into Python code later.

Screenshot 2

Curl command for the first request is:

curl 'https://ahrefs.com/v4/ftBacklinkCheckerPrimary' \
  -H 'authority: ahrefs.com' \
  -H 'accept: */*' \
  -H 'accept-language: en-GB,en-US;q=0.9,en;q=0.8' \
  -H 'cache-control: no-cache' \
  -H 'content-type: application/json; charset=utf-8' \
  -H 'cookie: __cf_bm=n2OYqOmarAhgWDxD3Tad6v1BAgLYFTtI3J8RxY9yapc-1687855654-0-AaF8nuZhwZT9A7zWj+Ojn1N58AhPC9jyUi+3Mfbpk/3pXYXEfdmpahcomXy/Bjv/TF6Qu3agnbRSquDnmjFC9RvKXWdwH50W/qipBYlv9DsD' \
  -H 'origin: https://ahrefs.com' \
  -H 'pragma: no-cache' \
  -H 'referer: https://ahrefs.com/backlink-checker' \
  -H 'sec-ch-ua: "Not.A/Brand";v="8", "Chromium";v="114", "Google Chrome";v="114"' \
  -H 'sec-ch-ua-mobile: ?0' \
  -H 'sec-ch-ua-platform: "macOS"' \
  -H 'sec-fetch-dest: empty' \
  -H 'sec-fetch-mode: cors' \
  -H 'sec-fetch-site: same-origin' \
  -H 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36' \
  --data-raw '{"captcha":"0.wsyuoTAQbPdempTdrWisjoNqxj6LRJwGyP4nnYteWF9mHnSQZ-v5ofEyoc1MVEMuUz-N04E5CpdhwBaY9tPpAbDxNjnLKqdvF54VxppzncFjDg7JEbS3KSFxL-9Zkzk_lBx9DiuID2Ojn8s5N7X2BUCMtITXLqHt5kdJhgYphmtRHB4F8w6zXXaqi7830p1pKD17lRxiKIqCTJowy6Q5J-0vx-OqWtN1NN6sv6uGLBvOhyWVcxPO3GGSjnxisXwF5Y2Tzjbr7gGjnOrePZH2adWbJlxFbgQ0gyT-qul6dR7xq6-2IYUPp8ud0ZJ9qomq2xJinE5d_TCF_4is-5kpW4ovGBFbreotGh5sR_6vAEn2nq-SjovnxMya0fA9XreX5lUtU3yxymOa7BWWYZoQJg.s2MS2XLcbogodDf16-NyPA.1b163e522a4349cf7556af8134812e0f3e9d53c0dca934b6a60010e558162db8","mode":"exact","url":"https://ahrefs.com/backlink-checker"}' \
  --compressed

In HTTP request payload there’s JSON object with captcha token and URL the user provided.

The JSON string in a response is of following form (reformatted for readability):

[
    "Ok",
    {
        "data": {
            "domainRating": 91.0,
            "urlRating": 41,
            "backlinks": 29409,
            "refdomains": 4917,
            "dofollowBacklinks": 49,
            "dofollowRefdomains": 67
        },
        "signedInput": {
            "input": {
                "url": "ahrefs.com/backlink-checker",
                "mode": "exact",
                "validUntil": "2023-06-27T20:48:03Z"
            },
            "signature": "ed9ead2b3e58fc3f45374cd562c51c5ba563d560206d9870b4d7695ff5a14652"
        }
    }
]

This provides a status (Ok) and some SEO-related numbers about the URL/domain, including a number of backlinks. If it was zero we could quit the API flow at this point.

The signedInput object goes into the second API request:

curl 'https://ahrefs.com/v4/ftBacklinkCheckerSecondary' \
  -H 'authority: ahrefs.com' \
  -H 'accept: */*' \
  -H 'accept-language: en-GB,en-US;q=0.9,en;q=0.8' \
  -H 'cache-control: no-cache' \
  -H 'content-type: application/json; charset=utf-8' \
  -H 'cookie: __cf_bm=n2OYqOmarAhgWDxD3Tad6v1BAgLYFTtI3J8RxY9yapc-1687855654-0-AaF8nuZhwZT9A7zWj+Ojn1N58AhPC9jyUi+3Mfbpk/3pXYXEfdmpahcomXy/Bjv/TF6Qu3agnbRSquDnmjFC9RvKXWdwH50W/qipBYlv9DsD' \
  -H 'origin: https://ahrefs.com' \
  -H 'pragma: no-cache' \
  -H 'referer: https://ahrefs.com/backlink-checker' \
  -H 'sec-ch-ua: "Not.A/Brand";v="8", "Chromium";v="114", "Google Chrome";v="114"' \
  -H 'sec-ch-ua-mobile: ?0' \
  -H 'sec-ch-ua-platform: "macOS"' \
  -H 'sec-fetch-dest: empty' \
  -H 'sec-fetch-mode: cors' \
  -H 'sec-fetch-site: same-origin' \
  -H 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36' \
  --data-raw '{"reportType":"TopBacklinks","signedInput":{"signature":"ed9ead2b3e58fc3f45374cd562c51c5ba563d560206d9870b4d7695ff5a14652","input":{"validUntil":"2023-06-27T20:48:03.000Z","mode":"exact","url":"ahrefs.com/backlink-checker"}}}' \
  --compressed

The latter API call yields the data we can extract and save somewhere. The JSON string in the response is something like this:

[
    "TopBacklinks",
    {
        "topBacklinks": {
            "backlinks": [
                {
                    "anchor": "Ahrefs’ Backlink Checker",
                    "domainRating": 91,
                    "edu": false,
                    "gov": false,
                    "httpCode": 200,
                    "redirectChain": [
                        {
                            "url": "https://ahrefs.com/backlink-checker?ref=buffer.com",
                            "urlNew": "None",
                            "redirectCode": 0,
                            "mustBeGray": false,
                            "httpCodeCross": false,
                            "chainDestinationChanged": false,
                            "hasDelReason": false,
                            "delReasonTitle": "None",
                            "delReasonKey": "None",
                            "lostRedirectReason": "None",
                            "isLost": true
                        }
                    ],
                    "text": true,
                    "textPost": "",
                    "textPre": "6.",
                    "title": "27 Simple and Free SEO Tools [Updated for 2023]",
                    "urlFrom": "https://buffer.com/library/free-seo-tools/",
                    "urlTo": "https://ahrefs.com/backlink-checker",
                    "inRendered": true,
                    "inRaw": true,
                    "redirectCode": 0,
                    "original": true,
                    "urlToMustBeGray": false,
                    "urlToHttpCodeCross": false,
                    "urlToNew": "None",
                    "urlToChainDestinationChanged": false,
                    "urlToDelReasonTitle": "None",
                    "urlToDelReasonKey": "None",
                    "urlToHasDelReason": false,
                    "lost_redirect_reason": "",
                    "del_reason": "",
                    "lost_redirect_source": "",
                    "lost_redirect_new_target": "",
                    "isLost": true
                },
                ...

            ],
            "total": 100
        }
    }
]

Screenshot 3

Also notice that API calls made from client side JS code have __cf_bm cookie added to them. This suggests that at least one of Cloudflare’s antibot products aside from Turnstile is deployed.

Since Ahrefs Backlink Checker is a growth tool meant to provide a potential user of Ahrefs a taste of what would be possible upon becoming a paying customer, the result list is limited to 100 entries. But that is not a problem for use cases involving exploration of obscure corners of the web.

So we want to develop the script now. But Ahrefs is using at least one feature of Cloudflare CDN that is meant to prevent or at least hinder the exact thing we want to do now. So I did some experimentation to get the details right:

For Turnstile part, both proxied and proxyless solutions from Anti-captcha work fine. Since proxy is not needed in this case, I chose a proxyless solution.
There is no need to talk HTTP/2, but TLS fingerprint has to be normalized to match the one from real browser.
We don’t need to worry about __cf_bm cookie, as it does not seem to matter much. First step of API flow works properly once we have TLS fingerprint normalised and captcha solved.
No proxies are needed as generating requests from DC IP did not cause any problems.

The complete script that does backlink scraping for a given list of pages is as follows:

#!/usr/bin/python3

import csv
from pprint import pprint
import os
import sys

from anticaptchaofficial.turnstileproxyless import *
import requests
import tls_client

FIELDNAMES = ["url", "backlink_url", "anchor", "page_title"]


def create_solver():
    solver = turnstileProxyless()

    solver.set_verbose(1)
    solver.set_key(os.getenv("ANTICAPTCHA_KEY"))
    solver.set_website_url("https://ahrefs.com/backlink-checker")
    solver.set_website_key("0x4AAAAAAAAzi9ITzSN9xKMi")

    return solver


def get_turnstile_token(solver):
    token = solver.solve_and_return_solution()
    if token != 0:
        print(token)
        return token
    else:
        print("task finished with error " + solver.error_code)

    return None


def create_session():
    session = tls_client.Session(
        client_identifier="chrome112", random_tls_extension_order=True
    )

    session.headers = {
        "authority": "ahrefs.com",
        "accept": "*/*",
        "accept-language": "en-GB,en-US;q=0.9,en;q=0.8",
        "cache-control": "no-cache",
        "content-type": "application/json; charset=utf-8",
        "origin": "https://ahrefs.com",
        "pragma": "no-cache",
        "referer": "https://ahrefs.com/backlink-checker",
        "sec-ch-ua": '"Google Chrome";v="113", "Chromium";v="113", "Not-A.Brand";v="24"',
        "sec-ch-ua-mobile": "?0",
        "sec-ch-ua-platform": '"macOS"',
        "sec-fetch-dest": "empty",
        "sec-fetch-mode": "cors",
        "sec-fetch-site": "same-origin",
        "user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36",
    }

    resp = session.get("https://ahrefs.com/backlink-checker")
    print(resp)

    return session


def get_backlinks(session, solver, url):
    token = get_turnstile_token(solver)
    if token is None:
        print("Failed to get turnstile token - retrying")
        token = get_turnstile_token(solver)

    if token is None:
        return

    json_data = {"captcha": token, "mode": "exact", "url": url}

    resp1 = session.post(
        "https://ahrefs.com/v4/ftBacklinkCheckerPrimary", json=json_data
    )
    print(resp1.url)
    print(resp1.text)
    pprint(resp1.json())

    json_arr = resp1.json()

    if len(json_arr) != 2:
        return None

    n_backlinks = json_arr[1].get("data", dict()).get("backlinks")
    if n_backlinks is None or n_backlinks == 0:
        return None

    signed_input = json_arr[1].get("signedInput")

    json_data2 = {"reportType": "TopBacklinks", "signedInput": signed_input}

    resp2 = session.post(
        "https://ahrefs.com/v4/ftBacklinkCheckerSecondary", json=json_data2
    )
    print(resp2.url)
    pprint(resp2.json())

    if len(resp2.json()) != 2:
        return None

    return resp2.json()[1].get("topBacklinks", dict()).get("backlinks")


def main():
    if len(sys.argv) != 3:
        print("Usage:")
        print("{} <url_list_file> <out_file>".format(sys.argv[0]))
        return

    url_list_file = sys.argv[1]
    out_file = sys.argv[2]

    solver = create_solver()
    session = create_session()

    in_f = open(url_list_file, "r")
    urls = in_f.read().strip().split("\n")
    in_f.close()

    out_f = open(out_file, "w", encoding="utf-8")

    csv_writer = csv.DictWriter(out_f, fieldnames=FIELDNAMES, lineterminator="\n")
    csv_writer.writeheader()

    for url in urls:
        url = url.strip()

        try:
            raw_backlinks = get_backlinks(session, solver, url)
        except Exception as e:
            print(e)
            continue

        if raw_backlinks is None:
            continue

        for backlink_dict in raw_backlinks:
            backlink_url = backlink_dict.get("urlFrom")
            anchor = backlink_dict.get("anchor")
            page_title = backlink_dict.get("title")

            row = {
                "url": url,
                "backlink_url": backlink_url,
                "anchor": anchor,
                "page_title": page_title,
            }

            pprint(row)
            csv_writer.writerow(row)

    out_f.close()


if __name__ == "__main__":
    main()

Results are saved into CSV file:

$ head res.csv 
url,backlink_url,anchor,page_title
https://www.trickster.dev/post/dont-jsfuck-with-me-part-3/,https://geek.ds3783.com/2023/06/%E5%88%AB%E8%B7%9F%E6%88%91%E6%9D%A5jsfuck%EF%BC%9A%E7%AC%AC%E4%B8%89%E9%83%A8%E5%88%86/,详情参考,别跟我来JSFuck：第三部分 - 偏执的码农
https://www.trickster.dev/post/prompting-the-llm-is-no-substitute-for-knowledge-work/,https://www.theteams.kr/stack/news_view/TrackJS/10,Prompting the LLM is no substitute for knowledge work 2023-05-31,TrackJS 해외 소식 | 더팀스
https://www.trickster.dev/post/prompting-the-llm-is-no-substitute-for-knowledge-work/,https://hacker.martintudor.com/user?id=rl1987,Prompting the LLM is no substitute for knowledge work,Hacker news
https://www.trickster.dev/post/prompting-the-llm-is-no-substitute-for-knowledge-work/,https://geek.ds3783.com/2023/05/%E6%BF%80%E5%8A%B1llm%E4%B8%8D%E8%83%BD%E6%9B%BF%E4%BB%A3%E7%9F%A5%E8%AF%86%E5%B7%A5%E4%BD%9C%E3%80%82/,详情参考,激励LLM不能替代知识工作。 - 偏执的码农
https://www.trickster.dev/post/understanding-javascript-packers/,https://www.theteams.kr/stack/news_view/Packet/4,Understanding JavaScript Packers 2023-05-30,Packet 해외 소식 | 더팀스
https://www.trickster.dev/post/understanding-javascript-packers/,https://geek.ds3783.com/2023/05/%E7%90%86%E8%A7%A3javascript%E6%89%93%E5%8C%85%E5%99%A8/,详情参考,理解JavaScript打包器 - 偏执的码农
https://www.trickster.dev/post/understanding-javascript-packers/,https://hacker.martintudor.com/user?id=rl1987,Understanding JavaScript Packers,Hacker news
https://www.trickster.dev/post/understanding-javascript-packers/,https://lampfull.com/news.php,Understanding JavaScript Packers,News - Gliath
https://www.trickster.dev/post/understanding-javascript-packers/,https://lr.relyma.com/r/javascript,trickster.dev,𝚓𝚊𝚟𝚊𝚜𝚌𝚛𝚒𝚙𝚝

Screenshot 4

Don't JSFuck with me: Part 3

Sun, 25 Jun 2023 00:00:00 +0000

Previously on Trickster Dev:

By using AST transforms developed so far, JSFuck output for character @ can be simplified to:

[]["flat"]["constructor"]("return\"" + ([]["flat"]["constructor"]("return/false/")()["constructor"]("/") + [])[1] + [1] + [0] + [0] + "\"")();

What about other characters that have null value in the MAPPING object in jsfuck.js?

For H we have:

[]["flat"]["constructor"]("return\"" + ([]["flat"]["constructor"]("return/false/")()["constructor"]("/") + [])[1] + [1] + [1] + [0] + "\"")();

For J we have:

[]["flat"]["constructor"]("return\"" + ([]["flat"]["constructor"]("return/false/")()["constructor"]("/") + [])[1] + [1] + [1] + [2] + "\"")();

For K we have:

[]["flat"]["constructor"]("return\"" + ([]["flat"]["constructor"]("return/false/")()["constructor"]("/") + [])[1] + [1] + [1] + [3] + "\"")();

For ~ we have:

[]["flat"]["constructor"]("return\"" + ([]["flat"]["constructor"]("return/false/")()["constructor"]("/") + [])[1] + [1] + [7] + [6] + "\"")();

The code we end up is very similar and follows a clear pattern.

Again, we can use NodeJS REPL to verify that this crazy shebang evaluates back to the original character:

> []["flat"]["constructor"]("return\"" + ([]["flat"]["constructor"]("return/false/")()["constructor"]("/") + [])[1] + [1] + [7] + [6] + "\"")();
'~'

Let us try to pick this stuff apart.

Due to type coercion, [1] + [7] + [6] part can be simplified to a single numeric string:

> [1] + [7] + [6]
'176'

Now what can we learn about code deeper in the statement?

> []["flat"]["constructor"]("return/false/")()
/false/
> []["flat"]["constructor"]("return/false/")()["constructor"]
[Function: RegExp]
> []["flat"]["constructor"]("return/false/")()["constructor"]("/")
/\//
> []["flat"]["constructor"]("return/false/")()["constructor"]("/") + []
'/\\//'
> ([]["flat"]["constructor"]("return/false/")()["constructor"]("/") + [])[1]
'\\'

This statement creates a regular expression object (/false/), accesses it’s contructor property to get a corresponding constructor function object (RegExp()), calls it with parameter / (this results in new regular expression object), then add an empty array to a result to stringify it through type coercion. Resulting string is indexed to to extract a single backslash character.

Based on this, we can perform some manual simplifications to this code:

> []["flat"]["constructor"]("return\"" + ([]["flat"]["constructor"]("return/false/")()["constructor"]("/") + [])[1] + '176' + "\"")();
'~'
> []["flat"]["constructor"]("return\"" + (RegExp("/") + [])[1] + '176' + "\"")();
'~'
> []["flat"]["constructor"]("return\"" + '\\' + '176' + "\"")();
'~'
> []["flat"]["constructor"]("return\"\\176\"")();
'~'

What is the number 176 doing here? In the ASCII standard, octal 176 corresponds to character ~ (see ascii(7) manpage). After undoing this octal-encoding, code gets even simpler:

> []["flat"]["constructor"]("return\"~\"")();
'~'

But what about this []["flat"]["constructor"] part we have here? JSFuck web app front page notes it is equivalent to calling eval(). Let us verify that for ourselves.

> []["flat"]
[Function: flat]
> typeof []["flat"]
'function'
> []["flat"]["constructor"]
[Function: Function]
> typeof []["flat"]["constructor"]
'function'
> []["flat"]["constructor"]("console.log(1)")
[Function: anonymous]
> []["flat"]["constructor"]("console.log(1)")()
1
undefined
> []["flat"]["constructor"] == Function
true
> Function("console.log(1)")()
1
undefined

So let us try to refactor this code bit by bit using Babel AST transformations.

First transformation is for converting []["flat"]["constructor"]("return/false/")() into /false/:

export default function (babel) {
  const { types: t } = babel;

  return {
    name: "simplify-false-regex", // not required
    visitor: {
      CallExpression(path) {
		const callExpr = path.node;
        if (callExpr.arguments.length != 0) return;
        if (!t.isCallExpression(callExpr.callee)) return;
        const calleeMemberExpr = callExpr.callee.callee;
        if (!t.isMemberExpression(calleeMemberExpr)) return;
        const memberExpr2 = calleeMemberExpr.object;
        if (!t.isMemberExpression(memberExpr2)) return;
        const arrayObj = memberExpr2.object;
        if (!t.isArrayExpression(arrayObj)) return;
        if (arrayObj.elements.length != 0) return;
        const property1 = memberExpr2.property;
        if (!t.isStringLiteral(property1)) return;
        if (property1.value != "flat") return;
        const property2 = calleeMemberExpr.property;
        if (!t.isStringLiteral(property2)) return;
        if (property2.value != "constructor") return;
        if (callExpr.callee.arguments.length != 1) return;
        const argStrLiteral = callExpr.callee.arguments[0];
        if (!t.isStringLiteral(argStrLiteral)) return;
        if (argStrLiteral.value != "return/false/") return;      
        path.replaceWith(t.regExpLiteral('false', ''));
      }
    }
  };
}

Now we need to deal with regexp constructor thing that accepts a string argument. There’s another transform for that:

export default function (babel) {
  const { types: t } = babel;

  return {
    name: "refactor-regex-constr", // not required
    visitor: {
      CallExpression(path) {
        const callExpr = path.node;
        if (callExpr.arguments.length != 1) return;
        const argument = callExpr.arguments[0];
        if (!t.isStringLiteral(argument)) return;
        const calleeMemberExpr = callExpr.callee;
        if (!t.isMemberExpression(calleeMemberExpr)) return;
        if (!t.isRegExpLiteral(calleeMemberExpr.object)) return;
        let property = calleeMemberExpr.property;
        if (!t.isStringLiteral(property)) return;
        if (property.value != "constructor") return;
        
        path.replaceWith(
          t.regExpLiteral(argument.value.replace('/', '\\/'), '')
        );
      }
    }
  };
}

Now we deal with addition of regular expression with empty array to stringify the regular expression object. Current code from part 1 deals with similar stuff, but is unable to cover what we have now. Thus we write one more small transformation to only cover this case:

export default function (babel) {
  const { types: t } = babel;

  return {
    name: "regex-str", // not required
    visitor: {
      BinaryExpression(path) {
        const binExpr = path.node;
        const left = binExpr.left;
        if (!t.isRegExpLiteral(left)) return;
        const right = binExpr.right;
        if (!t.isArrayExpression(right)) return;
        if (right.elements.length != 0) return;
        
        const regexStr = String(RegExp(left.pattern));
        
        path.replaceWith(t.stringLiteral(regexStr));
      }
    }
  };
}

To convert []["flat"]["constructor"]() into eval() calls we develop the following transform:

export default function (babel) {
  const { types: t } = babel;

  return {
    name: "fix-eval", // not required
    visitor: {
      CallExpression(path) {
        let node = path.node;
        if (node.arguments.length != 1) return;
        if (!t.isStringLiteral(node.arguments[0])) return;
        let parent = path.parent;
        if (!t.isCallExpression(parent)) return;
        let callee = node.callee;
        if (!t.isMemberExpression(callee)) return;
        if (!t.isStringLiteral(callee.property)) return;
        let key1 = callee.property.value;
        if (!t.isMemberExpression(callee.object)) return;
        if (!t.isStringLiteral(callee.object.property)) return;
        let key2 = callee.object.property.value;
        if (key1 === "constructor" && (key2 === "filter" || key2 === "flat")) {
          let evalCallExpr = t.callExpression(t.identifier('eval'), node.arguments);
          path.parentPath.replaceWith(evalCallExpr);
        }
      }
    }
  };
}

To convert string-returning eval() calls into string literals we do one more transformation:

export default function (babel) {
  const { types: t } = babel;

  return {
    name: "eval-return-str", // not required
    visitor: {
      CallExpression(path) {
        let node = path.node;
        if (!t.isIdentifier(node.callee)) return;
        if (node.callee.name != "eval") return;
        if (node.arguments.length != 1) return;
        if (!t.isStringLiteral(node.arguments[0])) return;
        
        let argStr = node.arguments[0].value;
        
        if (argStr.startsWith("return\"") && argStr.endsWith("\"")) {
          argStr = argStr.substr("return\"".length);
          argStr = argStr.substr(0, argStr.length-1);
          path.replaceWith(t.stringLiteral(argStr));
        }
      }
    }
  };
}

During this deobfuscation exercise, some more AST transforms were developed to cover new edge cases as they emerge. It would be excessive to cover them all in this post, as they all follow the pattern of matching the subtree for code we want to change and modifying to simplify it.

At this point, we can talk about unifying all the code we have into deobfuscator program that can be launched from a command line. Each AST transform we prototyped in AST Explorer can be trivially converted into Babel plugin by changing couple lines in the code. For example, the undo-string-trick transform becomes the following Babel plugin in file plugin_undo_string_trick.js:

const babel = require("@babel/core");
const t = require("@babel/types");

const logASTChange = require("./debug.js").logASTChange;

module.exports = function (babel) {
  return {
    name: "undo-string-trick", // not required
    visitor: {
      Identifier(path) {
        let node = path.node;
       	if (!t.isBinaryExpression(path.parent)) return;
        if (path.parent.operator != "+") return;
        if (node.name === "String") {
          let newNode = t.valueToNode(String(String));
          logASTChange("undo-string-trick", node, newNode);
          path.replaceWith(newNode);
        }
      }
    }
  };
}

You may notice that logASTChange() function is being imported from debug.js file. This function is meant to show us how the code is being changed as the transform is applied, so that we could see a step-by-step progress in the standard output. The code in debug.js file is as follows:

const generate = require("@babel/generator").default;

function logASTChange(pluginName, ast1, ast2) {
  const js1 = generate(ast1).code;
  const js2 = generate(ast2).code;

  console.log("PLUGIN NAME:", pluginName);
  console.log("FROM CODE:", js1);
  console.log("TO CODE:", js2);
}

module.exports = { logASTChange };

Code before and after the modification is being regenerated from AST form and printed into standard output with the Babel plugin name.

In app.js file we have code that unites all the plugins into single program with basic CLI:

const fs = require("fs");
const path = require("path");
const process = require("process");

const babel = require("@babel/core");
const t = require("@babel/types");

if (process.argv.length != 4) {
  console.log("Usage:");
  console.log("node app.js <input_js> <output_js>");
  process.exit(0);
}

const input_js = process.argv[2];
const output_js = process.argv[3];

let js = fs.readFileSync(input_js, "utf-8");
console.log(js);
console.log("-----------------------------------------------------------------");

while (true) {
  const result = babel.transformSync(js, {
      plugins: [
        path.join(__dirname, 'plugin_eval_expr.js'),
        path.join(__dirname, 'plugin_undo_flat_trick.js'),
        path.join(__dirname, 'plugin_undo_entries_trick.js'),
        path.join(__dirname, 'plugin_undo_concat_trick.js'),
        path.join(__dirname, 'plugin_undo_slice_trick.js'),
        path.join(__dirname, 'plugin_undo_boolean_trick.js'),
        path.join(__dirname, 'plugin_undo_fontcolor_trick.js'),
        path.join(__dirname, 'plugin_undo_italics_trick.js'),
        path.join(__dirname, 'plugin_undo_function_escape_trick.js'),
        path.join(__dirname, 'plugin_undo_escape_call.js'),
        path.join(__dirname, 'plugin_undo_function_date_trick.js'),
        path.join(__dirname, 'plugin_undo_regexp_trick.js'),
        path.join(__dirname, 'plugin_undo_string_trick.js'),
        path.join(__dirname, 'plugin_undo_number_tostring_trick.js'),
        path.join(__dirname, 'plugin_undo_object_tostring_trick.js'),
        path.join(__dirname, 'plugin_eval_return_str.js'),
        path.join(__dirname, 'plugin_eval_return_fncall.js'),
        path.join(__dirname, 'plugin_fix_eval.js'),
        path.join(__dirname, 'plugin_constructor_str.js'),
        path.join(__dirname, 'plugin_refactor_regex_constr.js'),
        path.join(__dirname, 'plugin_regex_str.js'),
        path.join(__dirname, 'plugin_simplify_false_regex.js'),
        path.join(__dirname, 'plugin_string_array_join.js'),
        path.join(__dirname, 'plugin_string_constructor_name.js'),
        path.join(__dirname, 'plugin_string_split.js'),
        path.join(__dirname, 'plugin_eval_program.js')
      ]
  });

  if (result.code == js) break;

  console.log(result.code);
  console.log("-----------------------------------------------------------------");

  js = result.code;
}

fs.writeFileSync(path.join(__dirname, output_js), js, 'utf-8');

Like we did manually with AST Explorer, this program modifies the input code until it no longer changes.

As a sidenote, undo-entries-trick transform from earlier had to be modified to prevent it from breaking the code:

-          path.replaceWith(t.stringLiteral(String(Array.prototype.entries)));
+          path.replaceWith(t.stringLiteral(String(String([]["entries"]()))));

Furthermore, code in eval-return-str transform was extended to support escaped characters in octal/hexadecimal form.

The following listing provides an example of debug output from deobfuscator (most lines were removed for brevity):

$ node app.js examples/in1.js examples/out1.js
[][(![]+[])[+[]]+(![]+[])[!+[]+!+[]]+(![]+[])[+!+[]]+(!![]+[])[+[]]][([][(![]+[])[+[]]+(![]+[])[!+[]+!+[]]+(![]+[])[+!+[]]+(!![]+[])[+[]]]+[])[!+[]+!+[]+!+[]]+(!![]+[][(![]+[])[+[]]+(![]+[])[!+[]+!+[]]+(![]+[])[+!+[]]+(!![]+[])[+[]]])[+!+[]+[+[]]]+([][[]]+[])[+!+[]]+(![]+[])[!+[]+!+[]+!+[]]+(!![]+[])[+[]]+(!![]+[])[+!+[]]+([][[]]+[])[+[]]+([][(![]+[])[+[]]+(![]+[])[!+[]+!+[]]+(![]+[])[+!+[]]+(!![]+[])[+[]]]+[])[!+[]+!+[]+!+[]]+(!![]+[])[+[]]+(!![]+[][(![]+[])[+[]]+(![]+[])[!+[]+!+[]]+(![]+[])[+!+[]]+(!![]+[])[+[]]])[+!+[]+[+[]]]+(!![]+[])[+!+[]]]([][(![]+[])[+[]]+(![]+[])[!+[]+!+[]]+(![]+[])[+!+[]]+(!![]+[])[+[]]][([][(![]+[])[+[]]+(![]+[])[!+[]+!+[]]+(![]+[])[+!+[]]+(!![]+[])[+[]]]+[])[!+[]+!+[]+!+[]]+(!![]+[][(![]+[])[+[]]+(![]+[])[!+[]+!+[]]+(![]+[])[+!+[]]+(!![]+[])[+[]]])[+!+[]+[+[]]]+([][[]]+[])[+!+[]]+(![]+[])[!+[]+!+[]+!+[]]+(!![]+[])[+[]]+(!![]+[])[+!+[]]+([][[]]+[])[+[]]+([][(![]+[])[+[]]+(![]+[])[!+[]+!+[]]+(![]+[])[+!+[]]+(!![]+[])[+[]]]+[])[!+[]+!+[]+!+[]]+(!![]+[])[+[]]+(!![]+[][(![]+[])[+[]]+(![]+[])[!+[]+!+[]]+(![]+[])[+!+[]]+(!![]+[])[+[]]])[+!+[]+[+[]]]+(!![]+[])[+!+[]]]((!![]+[])[+!+[]]+(!![]+[])[!+[]+!+[]+!+[]]+(!![]+[])[+[]]+([][[]]+[])[+[]]+(!![]+[])[+!+[]]+([][[]]+[])[+!+[]]+([]+[])[(![]+[])[+[]]+(!![]+[][(![]+[])[+[]]+(![]+[])[!+[]+!+[]]+(![]+[])[+!+[]]+(!![]+[])[+[]]])[+!+[]+[+[]]]+([][[]]+[])[+!+[]]+(!![]+[])[+[]]+([][(![]+[])[+[]]+(![]+[])[!+[]+!+[]]+(![]+[])[+!+[]]+(!![]+[])[+[]]]+[])[!+[]+!+[]+!+[]]+(!![]+[][(![]+[])[+[]]+(![]+[])[!+[]+!+[]]+(![]+[])[+!+[]]+(!![]+[])[+[]]])[+!+[]+[+[]]]+(![]+[])[!+[]+!+[]]+(!![]+[][(![]+[])[+[]]+(![]+[])[!+[]+!+[]]+(![]+[])[+!+[]]+(!![]+[])[+[]]])[+!+[]+[+[]]]+(!![]+[])[+!+[]]]()[+!+[]+[!+[]+!+[]]]+((![]+[])[+!+[]]+(![]+[])[!+[]+!+[]]+(!![]+[])[!+[]+!+[]+!+[]]+(!![]+[])[+!+[]]+(!![]+[])[+[]]+[+!+[]]+[!+[]+!+[]+!+[]+!+[]+!+[]+!+[]]+[!+[]+!+[]+!+[]+!+[]]+(!![]+[])[+[]]+[!+[]+!+[]+!+[]+!+[]+!+[]]+[+[]]+(!![]+[])[+[]]+[!+[]+!+[]+!+[]+!+[]]+[!+[]+!+[]]+(!![]+[])[+[]]+[+!+[]]+[+!+[]]+[+[]]+(!![]+[])[+[]]+[+!+[]]+[+!+[]]+[!+[]+!+[]]+(!![]+[])[+[]]+[+!+[]]+[+!+[]]+[!+[]+!+[]+!+[]]+(!![]+[])[+[]]+[+!+[]]+[+!+[]]+[!+[]+!+[]+!+[]+!+[]]+(!![]+[])[+[]]+[+!+[]]+[!+[]+!+[]]+[+[]]+(!![]+[])[+[]]+[+!+[]]+[!+[]+!+[]]+[+!+[]]+(!![]+[])[+[]]+[+!+[]]+[!+[]+!+[]]+[!+[]+!+[]+!+[]+!+[]+!+[]+!+[]]+(!![]+[])[+[]]+[+!+[]]+[!+[]+!+[]]+[!+[]+!+[]+!+[]+!+[]+!+[]+!+[]+!+[]]+(!![]+[])[+[]]+[+!+[]]+[!+[]+!+[]+!+[]]+[+[]]+(!![]+[])[+[]]+[+!+[]]+[!+[]+!+[]+!+[]]+[+!+[]]+(!![]+[])[+[]]+[+!+[]]+[!+[]+!+[]+!+[]]+[!+[]+!+[]]+(!![]+[])[+[]]+[!+[]+!+[]+!+[]+!+[]]+[+!+[]]+(!![]+[])[+[]]+[+!+[]]+[!+[]+!+[]+!+[]]+[!+[]+!+[]+!+[]+!+[]]+(!![]+[])[+[]]+[!+[]+!+[]+!+[]+!+[]]+[!+[]+!+[]+!+[]+!+[]+!+[]+!+[]+!+[]]+(!![]+[])[+[]]+[!+[]+!+[]+!+[]+!+[]]+[+!+[]]+(!![]+[])[+[]]+[!+[]+!+[]+!+[]+!+[]]+[!+[]+!+[]+!+[]]+(!![]+[])[+[]]+[!+[]+!+[]+!+[]+!+[]]+[!+[]+!+[]+!+[]+!+[]]+(!![]+[])[+[]]+[!+[]+!+[]+!+[]+!+[]+!+[]]+[!+[]+!+[]]+(!![]+[])[+[]]+[+!+[]]+[+[]]+[+[]]+(!![]+[])[+[]]+[+!+[]]+[!+[]+!+[]+!+[]]+[!+[]+!+[]+!+[]+!+[]+!+[]+!+[]]+(!![]+[])[+[]]+[+!+[]]+[!+[]+!+[]+!+[]]+[!+[]+!+[]+!+[]+!+[]+!+[]+!+[]+!+[]]+(!![]+[])[+[]]+[+!+[]]+[!+[]+!+[]+!+[]+!+[]]+[+[]]+(!![]+[])[+[]]+[+!+[]]+[!+[]+!+[]+!+[]+!+[]+!+[]+!+[]+!+[]]+[!+[]+!+[]+!+[]+!+[]]+(!![]+[])[+[]]+[+!+[]]+[!+[]+!+[]+!+[]+!+[]+!+[]+!+[]+!+[]]+[!+[]+!+[]+!+[]+!+[]+!+[]+!+[]]+(!![]+[])[+[]]+[!+[]+!+[]+!+[]+!+[]]+[!+[]+!+[]]+(!![]+[])[+[]]+[!+[]+!+[]+!+[]+!+[]+!+[]]+[+!+[]]+(!![]+[])[+[]]+[!+[]+!+[]+!+[]+!+[]+!+[]+!+[]+!+[]]+[!+[]+!+[]+!+[]])[(![]+[])[!+[]+!+[]+!+[]]+(+(!+[]+!+[]+[+!+[]]+[+!+[]]))[(!![]+[])[+[]]+(!![]+[][(![]+[])[+[]]+(![]+[])[!+[]+!+[]]+(![]+[])[+!+[]]+(!![]+[])[+[]]])[+!+[]+[+[]]]+([]+[])[([][(![]+[])[+[]]+(![]+[])[!+[]+!+[]]+(![]+[])[+!+[]]+(!![]+[])[+[]]]+[])[!+[]+!+[]+!+[]]+(!![]+[][(![]+[])[+[]]+(![]+[])[!+[]+!+[]]+(![]+[])[+!+[]]+(!![]+[])[+[]]])[+!+[]+[+[]]]+([][[]]+[])[+!+[]]+(![]+[])[!+[]+!+[]+!+[]]+(!![]+[])[+[]]+(!![]+[])[+!+[]]+([][[]]+[])[+[]]+([][(![]+[])[+[]]+(![]+[])[!+[]+!+[]]+(![]+[])[+!+[]]+(!![]+[])[+[]]]+[])[!+[]+!+[]+!+[]]+(!![]+[])[+[]]+(!![]+[][(![]+[])[+[]]+(![]+[])[!+[]+!+[]]+(![]+[])[+!+[]]+(!![]+[])[+[]]])[+!+[]+[+[]]]+(!![]+[])[+!+[]]][([][[]]+[])[+!+[]]+(![]+[])[+!+[]]+((+[])[([][(![]+[])[+[]]+(![]+[])[!+[]+!+[]]+(![]+[])[+!+[]]+(!![]+[])[+[]]]+[])[!+[]+!+[]+!+[]]+(!![]+[][(![]+[])[+[]]+(![]+[])[!+[]+!+[]]+(![]+[])[+!+[]]+(!![]+[])[+[]]])[+!+[]+[+[]]]+([][[]]+[])[+!+[]]+(![]+[])[!+[]+!+[]+!+[]]+(!![]+[])[+[]]+(!![]+[])[+!+[]]+([][[]]+[])[+[]]+([][(![]+[])[+[]]+(![]+[])[!+[]+!+[]]+(![]+[])[+!+[]]+(!![]+[])[+[]]]+[])[!+[]+!+[]+!+[]]+(!![]+[])[+[]]+(!![]+[][(![]+[])[+[]]+(![]+[])[!+[]+!+[]]+(![]+[])[+!+[]]+(!![]+[])[+[]]])[+!+[]+[+[]]]+(!![]+[])[+!+[]]]+[])[+!+[]+[+!+[]]]+(!![]+[])[!+[]+!+[]+!+[]]]](!+[]+!+[]+!+[]+[+!+[]])[+!+[]]+(![]+[])[!+[]+!+[]]+([![]]+[][[]])[+!+[]+[+[]]]+(!![]+[])[+[]]]((!![]+[])[+[]])[([][(!![]+[])[!+[]+!+[]+!+[]]+([][[]]+[])[+!+[]]+(!![]+[])[+[]]+(!![]+[])[+!+[]]+([![]]+[][[]])[+!+[]+[+[]]]+(!![]+[])[!+[]+!+[]+!+[]]+(![]+[])[!+[]+!+[]+!+[]]]()+[])[!+[]+!+[]+!+[]]+(!![]+[][(![]+[])[+[]]+(![]+[])[!+[]+!+[]]+(![]+[])[+!+[]]+(!![]+[])[+[]]])[+!+[]+[+[]]]+([![]]+[][[]])[+!+[]+[+[]]]+([][[]]+[])[+!+[]]](([][(![]+[])[+[]]+(![]+[])[!+[]+!+[]]+(![]+[])[+!+[]]+(!![]+[])[+[]]][([][(![]+[])[+[]]+(![]+[])[!+[]+!+[]]+(![]+[])[+!+[]]+(!![]+[])[+[]]]+[])[!+[]+!+[]+!+[]]+(!![]+[][(![]+[])[+[]]+(![]+[])[!+[]+!+[]]+(![]+[])[+!+[]]+(!![]+[])[+[]]])[+!+[]+[+[]]]+([][[]]+[])[+!+[]]+(![]+[])[!+[]+!+[]+!+[]]+(!![]+[])[+[]]+(!![]+[])[+!+[]]+([][[]]+[])[+[]]+([][(![]+[])[+[]]+(![]+[])[!+[]+!+[]]+(![]+[])[+!+[]]+(!![]+[])[+[]]]+[])[!+[]+!+[]+!+[]]+(!![]+[])[+[]]+(!![]+[][(![]+[])[+[]]+(![]+[])[!+[]+!+[]]+(![]+[])[+!+[]]+(!![]+[])[+[]]])[+!+[]+[+[]]]+(!![]+[])[+!+[]]]((!![]+[])[+!+[]]+(!![]+[])[!+[]+!+[]+!+[]]+(!![]+[])[+[]]+([][[]]+[])[+[]]+(!![]+[])[+!+[]]+([][[]]+[])[+!+[]]+(![]+[+[]])[([![]]+[][[]])[+!+[]+[+[]]]+(!![]+[])[+[]]+(![]+[])[+!+[]]+(![]+[])[!+[]+!+[]]+([![]]+[][[]])[+!+[]+[+[]]]+([][(![]+[])[+[]]+(![]+[])[!+[]+!+[]]+(![]+[])[+!+[]]+(!![]+[])[+[]]]+[])[!+[]+!+[]+!+[]]+(![]+[])[!+[]+!+[]+!+[]]]()[+!+[]+[+[]]]+![]+(![]+[+[]])[([![]]+[][[]])[+!+[]+[+[]]]+(!![]+[])[+[]]+(![]+[])[+!+[]]+(![]+[])[!+[]+!+[]]+([![]]+[][[]])[+!+[]+[+[]]]+([][(![]+[])[+[]]+(![]+[])[!+[]+!+[]]+(![]+[])[+!+[]]+(!![]+[])[+[]]]+[])[!+[]+!+[]+!+[]]+(![]+[])[!+[]+!+[]+!+[]]]()[+!+[]+[+[]]])()[([][(![]+[])[+[]]+(![]+[])[!+[]+!+[]]+(![]+[])[+!+[]]+(!![]+[])[+[]]]+[])[!+[]+!+[]+!+[]]+(!![]+[][(![]+[])[+[]]+(![]+[])[!+[]+!+[]]+(![]+[])[+!+[]]+(!![]+[])[+[]]])[+!+[]+[+[]]]+([][[]]+[])[+!+[]]+(![]+[])[!+[]+!+[]+!+[]]+(!![]+[])[+[]]+(!![]+[])[+!+[]]+([][[]]+[])[+[]]+([][(![]+[])[+[]]+(![]+[])[!+[]+!+[]]+(![]+[])[+!+[]]+(!![]+[])[+[]]]+[])[!+[]+!+[]+!+[]]+(!![]+[])[+[]]+(!![]+[][(![]+[])[+[]]+(![]+[])[!+[]+!+[]]+(![]+[])[+!+[]]+(!![]+[])[+[]]])[+!+[]+[+[]]]+(!![]+[])[+!+[]]]((![]+[+[]])[([![]]+[][[]])[+!+[]+[+[]]]+(!![]+[])[+[]]+(![]+[])[+!+[]]+(![]+[])[!+[]+!+[]]+([![]]+[][[]])[+!+[]+[+[]]]+([][(![]+[])[+[]]+(![]+[])[!+[]+!+[]]+(![]+[])[+!+[]]+(!![]+[])[+[]]]+[])[!+[]+!+[]+!+[]]+(![]+[])[!+[]+!+[]+!+[]]]()[+!+[]+[+[]]])+[])[+!+[]])+([]+[])[(![]+[])[+[]]+(!![]+[][(![]+[])[+[]]+(![]+[])[!+[]+!+[]]+(![]+[])[+!+[]]+(!![]+[])[+[]]])[+!+[]+[+[]]]+([][[]]+[])[+!+[]]+(!![]+[])[+[]]+([][(![]+[])[+[]]+(![]+[])[!+[]+!+[]]+(![]+[])[+!+[]]+(!![]+[])[+[]]]+[])[!+[]+!+[]+!+[]]+(!![]+[][(![]+[])[+[]]+(![]+[])[!+[]+!+[]]+(![]+[])[+!+[]]+(!![]+[])[+[]]])[+!+[]+[+[]]]+(![]+[])[!+[]+!+[]]+(!![]+[][(![]+[])[+[]]+(![]+[])[!+[]+!+[]]+(![]+[])[+!+[]]+(!![]+[])[+[]]])[+!+[]+[+[]]]+(!![]+[])[+!+[]]]()[+!+[]+[!+[]+!+[]]])())()
-----------------------------------------------------------------
PLUGIN NAME: eval-expr
FROM CODE: ![] + []
TO CODE: "false"
PLUGIN NAME: eval-expr
FROM CODE: +[]
TO CODE: 0
PLUGIN NAME: eval-expr
FROM CODE: ![] + []
TO CODE: "false"
PLUGIN NAME: eval-expr
FROM CODE: !+[] + !+[]
TO CODE: 2
PLUGIN NAME: eval-expr
FROM CODE: ![] + []
TO CODE: "false"
PLUGIN NAME: eval-expr
FROM CODE: +!+[]
TO CODE: 1
PLUGIN NAME: eval-expr
FROM CODE: !![] + []
TO CODE: "true"
PLUGIN NAME: eval-expr
FROM CODE: +[]
TO CODE: 0
PLUGIN NAME: eval-expr
FROM CODE: ![] + []
TO CODE: "false"
PLUGIN NAME: eval-expr
FROM CODE: +[]
TO CODE: 0
PLUGIN NAME: eval-expr
FROM CODE: ![] + []
TO CODE: "false"
PLUGIN NAME: eval-expr
FROM CODE: !+[] + !+[]
TO CODE: 2
PLUGIN NAME: eval-expr
FROM CODE: ![] + []
TO CODE: "false"
PLUGIN NAME: eval-expr
FROM CODE: +!+[]
TO CODE: 1
PLUGIN NAME: eval-expr
FROM CODE: !![] + []
TO CODE: "true"
PLUGIN NAME: eval-expr
FROM CODE: +[]
TO CODE: 0
PLUGIN NAME: eval-expr
FROM CODE: !+[] + !+[] + !+[]
TO CODE: 3
...
-----------------------------------------------------------------
PLUGIN NAME: undo-fontcolor-trick
FROM CODE: ""["fontcolor"]()
TO CODE: "<font color=\"undefined\"></font>"
PLUGIN NAME: constructor-str
FROM CODE: 0["constructor"]
TO CODE: "function Number() { [native code] }"
PLUGIN NAME: eval-expr
FROM CODE: "j" + "o" + "i" + "n"
TO CODE: "join"
PLUGIN NAME: undo-italics-trick
FROM CODE: "false0"["italics"]()
TO CODE: "<i>false0</i>"
PLUGIN NAME: undo-italics-trick
FROM CODE: "false0"["italics"]()
TO CODE: "<i>false0</i>"
PLUGIN NAME: undo-italics-trick
FROM CODE: "false0"["italics"]()
TO CODE: "<i>false0</i>"
PLUGIN NAME: undo-fontcolor-trick
FROM CODE: ""["fontcolor"]()
TO CODE: "<font color=\"undefined\"></font>"
[]["flat"]["constructor"]([]["flat"]["constructor"]("return" + "<font color=\"undefined\"></font>"[12] + "alert164t50t42t110t112t113t114t120t121t126t127t130t131t132t41t134t47t41t43t44t52t100t136t137t140t174t176t42t51t73"["s" + 211["to" + ""["constructor"]["na" + ("function Number() { [native code] }" + [])[11] + "e"]]("31")[1] + "l" + "i" + "t"]("t")["join"](([]["flat"]["constructor"]("return" + "<i>false0</i>"[10] + false + "<i>false0</i>"[10])()["constructor"]("<i>false0</i>"[10]) + [])[1]) + "<font color=\"undefined\"></font>"[12])())();
-----------------------------------------------------------------
PLUGIN NAME: eval-expr
FROM CODE: "<font color=\"undefined\"></font>"[12]
TO CODE: "\""
PLUGIN NAME: eval-expr
FROM CODE: "function Number() { [native code] }" + []
TO CODE: "function Number() { [native code] }"
PLUGIN NAME: eval-expr
FROM CODE: "<i>false0</i>"[10]
TO CODE: "/"
PLUGIN NAME: eval-expr
FROM CODE: "<i>false0</i>"[10]
TO CODE: "/"
PLUGIN NAME: eval-expr
FROM CODE: "<i>false0</i>"[10]
TO CODE: "/"
PLUGIN NAME: eval-expr
FROM CODE: "<font color=\"undefined\"></font>"[12]
TO CODE: "\""
[]["flat"]["constructor"]([]["flat"]["constructor"]("return" + "\"" + "alert164t50t42t110t112t113t114t120t121t126t127t130t131t132t41t134t47t41t43t44t52t100t136t137t140t174t176t42t51t73"["s" + 211["to" + ""["constructor"]["na" + "function Number() { [native code] }"[11] + "e"]]("31")[1] + "l" + "i" + "t"]("t")["join"](([]["flat"]["constructor"]("return" + "/" + false + "/")()["constructor"]("/") + [])[1]) + "\"")())();
-----------------------------------------------------------------
PLUGIN NAME: eval-expr
FROM CODE: "return" + "\""
TO CODE: "return\""
PLUGIN NAME: eval-expr
FROM CODE: "function Number() { [native code] }"[11]
TO CODE: "m"
PLUGIN NAME: eval-expr
FROM CODE: "return" + "/" + false + "/"
TO CODE: "return/false/"
[]["flat"]["constructor"]([]["flat"]["constructor"]("return\"" + "alert164t50t42t110t112t113t114t120t121t126t127t130t131t132t41t134t47t41t43t44t52t100t136t137t140t174t176t42t51t73"["s" + 211["to" + ""["constructor"]["na" + "m" + "e"]]("31")[1] + "l" + "i" + "t"]("t")["join"](([]["flat"]["constructor"]("return/false/")()["constructor"]("/") + [])[1]) + "\"")())();
-----------------------------------------------------------------
PLUGIN NAME: eval-expr
FROM CODE: "na" + "m" + "e"
TO CODE: "name"
PLUGIN NAME: simplify-false-regex
FROM CODE: []["flat"]["constructor"]("return/false/")()
TO CODE: /false/
[]["flat"]["constructor"]([]["flat"]["constructor"]("return\"" + "alert164t50t42t110t112t113t114t120t121t126t127t130t131t132t41t134t47t41t43t44t52t100t136t137t140t174t176t42t51t73"["s" + 211["to" + ""["constructor"]["name"]]("31")[1] + "l" + "i" + "t"]("t")["join"]((/false/["constructor"]("/") + [])[1]) + "\"")())();
-----------------------------------------------------------------
PLUGIN NAME: string-constructor-name
FROM CODE: ""["constructor"]["name"]
TO CODE: "String"
PLUGIN NAME: refactor-regex-constr
FROM CODE: /false/["constructor"]("/")
TO CODE: /\//
[]["flat"]["constructor"]([]["flat"]["constructor"]("return\"" + "alert164t50t42t110t112t113t114t120t121t126t127t130t131t132t41t134t47t41t43t44t52t100t136t137t140t174t176t42t51t73"["s" + 211["to" + "String"]("31")[1] + "l" + "i" + "t"]("t")["join"]((/\// + [])[1]) + "\"")())();
-----------------------------------------------------------------
PLUGIN NAME: eval-expr
FROM CODE: "to" + "String"
TO CODE: "toString"
PLUGIN NAME: regex-str
FROM CODE: /\// + []
TO CODE: "/\\//"
[]["flat"]["constructor"]([]["flat"]["constructor"]("return\"" + "alert164t50t42t110t112t113t114t120t121t126t127t130t131t132t41t134t47t41t43t44t52t100t136t137t140t174t176t42t51t73"["s" + 211["toString"]("31")[1] + "l" + "i" + "t"]("t")["join"]("/\\//"[1]) + "\"")())();
-----------------------------------------------------------------
PLUGIN NAME: undo-number-tostring-trick
FROM CODE: 211["toString"]("31")
TO CODE: "6p"
PLUGIN NAME: eval-expr
FROM CODE: "/\\//"[1]
TO CODE: "\\"
[]["flat"]["constructor"]([]["flat"]["constructor"]("return\"" + "alert164t50t42t110t112t113t114t120t121t126t127t130t131t132t41t134t47t41t43t44t52t100t136t137t140t174t176t42t51t73"["s" + "6p"[1] + "l" + "i" + "t"]("t")["join"]("\\") + "\"")())();
-----------------------------------------------------------------
PLUGIN NAME: eval-expr
FROM CODE: "6p"[1]
TO CODE: "p"
[]["flat"]["constructor"]([]["flat"]["constructor"]("return\"" + "alert164t50t42t110t112t113t114t120t121t126t127t130t131t132t41t134t47t41t43t44t52t100t136t137t140t174t176t42t51t73"["s" + "p" + "l" + "i" + "t"]("t")["join"]("\\") + "\"")())();
-----------------------------------------------------------------
PLUGIN NAME: eval-expr
FROM CODE: "s" + "p" + "l" + "i" + "t"
TO CODE: "split"
[]["flat"]["constructor"]([]["flat"]["constructor"]("return\"" + "alert164t50t42t110t112t113t114t120t121t126t127t130t131t132t41t134t47t41t43t44t52t100t136t137t140t174t176t42t51t73"["split"]("t")["join"]("\\") + "\"")())();
-----------------------------------------------------------------
PLUGIN NAME: string-split
FROM CODE: "alert164t50t42t110t112t113t114t120t121t126t127t130t131t132t41t134t47t41t43t44t52t100t136t137t140t174t176t42t51t73"["split"]("t")
TO CODE: ["aler", "164", "50", "42", "110", "112", "113", "114", "120", "121", "126", "127", "130", "131", "132", "41", "134", "47", "41", "43", "44", "52", "100", "136", "137", "140", "174", "176", "42", "51", "73"]
[]["flat"]["constructor"]([]["flat"]["constructor"]("return\"" + ["aler", "164", "50", "42", "110", "112", "113", "114", "120", "121", "126", "127", "130", "131", "132", "41", "134", "47", "41", "43", "44", "52", "100", "136", "137", "140", "174", "176", "42", "51", "73"]["join"]("\\") + "\"")())();
-----------------------------------------------------------------
PLUGIN NAME: string-array-join
FROM CODE: ["aler", "164", "50", "42", "110", "112", "113", "114", "120", "121", "126", "127", "130", "131", "132", "41", "134", "47", "41", "43", "44", "52", "100", "136", "137", "140", "174", "176", "42", "51", "73"]["join"]("\\")
TO CODE: "aler\\164\\50\\42\\110\\112\\113\\114\\120\\121\\126\\127\\130\\131\\132\\41\\134\\47\\41\\43\\44\\52\\100\\136\\137\\140\\174\\176\\42\\51\\73"
[]["flat"]["constructor"]([]["flat"]["constructor"]("return\"" + "aler\\164\\50\\42\\110\\112\\113\\114\\120\\121\\126\\127\\130\\131\\132\\41\\134\\47\\41\\43\\44\\52\\100\\136\\137\\140\\174\\176\\42\\51\\73" + "\"")())();
-----------------------------------------------------------------
PLUGIN NAME: eval-expr
FROM CODE: "return\"" + "aler\\164\\50\\42\\110\\112\\113\\114\\120\\121\\126\\127\\130\\131\\132\\41\\134\\47\\41\\43\\44\\52\\100\\136\\137\\140\\174\\176\\42\\51\\73" + "\""
TO CODE: "return\"aler\\164\\50\\42\\110\\112\\113\\114\\120\\121\\126\\127\\130\\131\\132\\41\\134\\47\\41\\43\\44\\52\\100\\136\\137\\140\\174\\176\\42\\51\\73\""
[]["flat"]["constructor"]([]["flat"]["constructor"]("return\"aler\\164\\50\\42\\110\\112\\113\\114\\120\\121\\126\\127\\130\\131\\132\\41\\134\\47\\41\\43\\44\\52\\100\\136\\137\\140\\174\\176\\42\\51\\73\"")())();
-----------------------------------------------------------------
PLUGIN NAME: fix-eval
FROM CODE: []["flat"]["constructor"]("return\"aler\\164\\50\\42\\110\\112\\113\\114\\120\\121\\126\\127\\130\\131\\132\\41\\134\\47\\41\\43\\44\\52\\100\\136\\137\\140\\174\\176\\42\\51\\73\"")
TO CODE: eval("return\"aler\\164\\50\\42\\110\\112\\113\\114\\120\\121\\126\\127\\130\\131\\132\\41\\134\\47\\41\\43\\44\\52\\100\\136\\137\\140\\174\\176\\42\\51\\73\"")
PLUGIN NAME: eval-return-str
FROM CODE: eval("return\"aler\\164\\50\\42\\110\\112\\113\\114\\120\\121\\126\\127\\130\\131\\132\\41\\134\\47\\41\\43\\44\\52\\100\\136\\137\\140\\174\\176\\42\\51\\73\"")
TO CODE: "alert(\"HJKLPQVWXYZ!\\'!#$*@^_`|~\");"
[]["flat"]["constructor"]("alert(\"HJKLPQVWXYZ!\\'!#$*@^_`|~\");")();
-----------------------------------------------------------------
PLUGIN NAME: fix-eval
FROM CODE: []["flat"]["constructor"]("alert(\"HJKLPQVWXYZ!\\'!#$*@^_`|~\");")
TO CODE: eval("alert(\"HJKLPQVWXYZ!\\'!#$*@^_`|~\");")
eval("alert(\"HJKLPQVWXYZ!\\'!#$*@^_`|~\");");
-----------------------------------------------------------------
PLUGIN NAME: eval-program
FROM CODE: eval("alert(\"HJKLPQVWXYZ!\\'!#$*@^_`|~\");");
TO CODE: alert("HJKLPQVWXYZ!\'!#$*@^_`|~");
alert("HJKLPQVWXYZ!\'!#$*@^_`|~");
-----------------------------------------------------------------

So the deobfuscator works for small snippets. The deobfuscated line is the same that I entered into JSFuck web app. However for bigger snippet there is a problem. During the AST traversal Babel runs out of stack space and crashes with the the following error message on the first iteration of the loop:

/Users/rl/src/trickster.dev-code/jsfuck_wip/jsfuck_deobfuscator/node_modules/@babel/traverse/lib/context.js:77
  visitSingle(node, key) {
             ^

RangeError: Maximum call stack size exceeded
    at TraversalContext.visitSingle (/Users/rl/src/trickster.dev-code/jsfuck_wip/jsfuck_deobfuscator/node_modules/@babel/traverse/lib/context.js:77:14)
    at TraversalContext.visit (/Users/rl/src/trickster.dev-code/jsfuck_wip/jsfuck_deobfuscator/node_modules/@babel/traverse/lib/context.js:133:19)
    at traverseNode (/Users/rl/src/trickster.dev-code/jsfuck_wip/jsfuck_deobfuscator/node_modules/@babel/traverse/lib/traverse-node.js:24:17)
    at NodePath.visit (/Users/rl/src/trickster.dev-code/jsfuck_wip/jsfuck_deobfuscator/node_modules/@babel/traverse/lib/path/context.js:107:52)
    at TraversalContext.visitQueue (/Users/rl/src/trickster.dev-code/jsfuck_wip/jsfuck_deobfuscator/node_modules/@babel/traverse/lib/context.js:105:16)
    at TraversalContext.visitSingle (/Users/rl/src/trickster.dev-code/jsfuck_wip/jsfuck_deobfuscator/node_modules/@babel/traverse/lib/context.js:79:19)
    at TraversalContext.visit (/Users/rl/src/trickster.dev-code/jsfuck_wip/jsfuck_deobfuscator/node_modules/@babel/traverse/lib/context.js:133:19)
    at traverseNode (/Users/rl/src/trickster.dev-code/jsfuck_wip/jsfuck_deobfuscator/node_modules/@babel/traverse/lib/traverse-node.js:24:17)
    at NodePath.visit (/Users/rl/src/trickster.dev-code/jsfuck_wip/jsfuck_deobfuscator/node_modules/@babel/traverse/lib/path/context.js:107:52)
    at TraversalContext.visitQueue (/Users/rl/src/trickster.dev-code/jsfuck_wip/jsfuck_deobfuscator/node_modules/@babel/traverse/lib/context.js:105:16)

Node.js v20.1.0

Notice function calls repeating in the stack trace. This means that we have recursion going on in here. JSFuck makes the AST excessively bloated and thus difficult to programmatically simplify with Babel.

I think this is the final point of my JSFuck deobfuscation journey. The original objective for doing this was to learn more about Babel and AST-level deobfuscation and that was achieved. The current deobfuscation approach was of some success in a way that it is able to simplify sufficiently small and simple snippets, but ran into a fundamental limitation of Babel (if you know how to solve this - kindly let me know).

Does that mean JSFuck output cannot be deobfuscated? No. There are open source examples that treat JS code as string to undo the changes that JSFuck is making:

JSUNFUck - C#/.NET program
jsunfuck.py file in cloudscraper Python module.
JSFuck Decoder - simple web app developed in vanilla JS/HTML.

Screenshot 1 Screenshot 2

This is indeed an easier and better way to deal with JSFuck being used to obfuscate client-side JavaScript code. But one cannot become stronger and smarter by always taking an easy way. We learned about JS type coercion and about various weird runtime tricks that JSFuck is relying on. Martin Kleppe, the developer of JSFuck is a smart, creative person and that shows in his work.

Prompting the LLM is no substitute for knowledge work

Wed, 31 May 2023 00:00:00 +0000

Artificial intelligence, especially large language models such as ChatGPT are all the rage now. ChatGPT has the distinction of being the fastest growing product in the history of technology - it reached the first one million users in just a few days. As such, it is the talk of the global village now. There is no shortage of overly excited people posting their takes on how it will change the world for the better and what you should be doing right now to benefit from that. On the other hand, some are worried about ending up jobless with no place to go in the dystopian reality of 2029. Who are right? We cannot tell without knowing the future, but I don’t think any of the extreme predictions are reasonable. In many cases they are not even intellectually honest (more on that later). I am not saying that LLMs are not a big deal. Building the computational machinery to make human language tractable by software has long been a Holy Grail of research fields such as computational linguistics and natural language processing. Making that work is a great technical achievement that should not be taken lightly. However I don’t believe in black-and-white thinking. I don’t even believe in grayscale thinking. There is way too much complexity in the current social, technical, political and cultural realities to be sufficiently captured by mere notion of “goodness”.

Can the LLMs make you jobless by text-generating your job position away? Such question was posed to sci-fi author Cory Doctorow in 2020, during the Remote Chaos Experience conference. Back then, ChatGPT was not released, but GPT-3 was already available. His response was that only people who have very boring, terrible jobs should be worried about GPT-3. That answer should not have been surprising to people who actually played a bit with GPT-3. It can certainly be guided into generating text that compares to what a human could write, but if left unattended it can easily go off the rails and start generating complete garbage.

ChatGPT (a.k.a. GPT-3.5-turbo) is not exactly immune to generating nonsense either. Let us do a quick experiment with aichat - one of the OpenAI API clients.

$ aichat
Welcome to aichat 0.8.0
Type ".help" for more information.
〉.model gpt-3.5-turbo

〉.prompt What online courses could be taken to learn JavaScript deobfuscation?

Ｐ〉
Ｐ〉;
There are a few online courses that could help you learn JavaScript deobfuscation:

1. "Advanced JavaScript Concepts" by Udemy: This course covers advanced JavaScript topics, including deobfuscation techniques.

2. "JavaScript Deobfuscation and Malware Analysis" by Pluralsight: This course focuses specifically on JavaScript deobfuscation and how to analyze malware that uses obfuscated JavaScript.

3. "Practical Malware Analysis" by the Malware Unicorn: This course covers malware analysis techniques, including how to deobfuscate JavaScript used in malware.

4. "JavaScript Deobfuscation for Web Developers" by Packt: This course focuses on deobfuscation techniques for JavaScript used in web development.

5. "Reverse Engineering Malware" by OpenSecurityTraining: This course covers a variety of reverse engineering techniques, including deobfuscating JavaScript used in malware.

We can see a factually wrong output here. Courses 2 and 4 don’t exist. Courses 1, 3 and 5 do exist, but they are not very relevant to JS deobfuscation. So we got an answer that is factually wrong, but written in a very confident wording. We got a problem of trust here. If we rely on ChatGPT to produce a text that is factually correct we cannot accept the possibility of some percentage of output being simply wrong. Thus it cannot be trusted to do all the work for us - to benefit from ChatGPT and similar systems, we have to work with them to achieve a desired result. That includes checking the generated text for factual correctness.

But we’re talking fairly obscure, niche stuff here. In fact, I kind of did it on purpose to demonstrate a major failure mode of LLMs. But what if ChatGPT was used to help with something more relevant to general society, such as preparing a lawsuit against an airline? One Steven Schwartz, an attorney with some three decades of experience, was using ChatGPT for (scary quotes) “research” when preparing paperwork to sue a certain airline over an injury allegedly sustained by a customer. That went horribly, scandalously wrong. Like in our quick experiment above, ChatGPT hallucinated references to prior legal cases and court decisions that simply did not happen. Schwartz himself got in trouble for submitting legal paperwork with false data and the story made it’s rounds across the press outlets. You know you really fuck up your work when your oopsie makes it’s way to New York Times.

GPT-4 is more advanced than ChatGPT, but is susceptible to hallucinating stuff as well (just ask the same question about deobfuscation courses). But what if there was a sufficiently advanced LLM that did not hallucinate and only generated text that was factually correct (unless specifically instructed to write fiction)? Realistically that is not even possible, as it relies on perfectly correct information being sourced for training of such model. However, for the sake of argument, let us assume that there is such a system running. If we have a perfect knowledge machine, can we use it to replace a human work? Well, the thing is, we still need to tell it what to do. Even a perfect LLM is GIGO: garbage in, garbage out. When it comes to non-trivial things (such as developing large software systems), describing them in sufficient detail in a way that captures all nuances and edge cases is quite difficult type of work. This can be seen in real world software projects, especially when they involve non-technical stakeholders. It’s naive for manager or entrepreneur to assume that soon enough they will be able to get rid of those pesky developers, as they will just tell a chatbot what code it should write. Quite likely the activity of getting an LLM to do things (generate or edit blog post, develop code, write marketing copy, etc.) will be new kind of work - prompt design/engineering.

Sure, you can use ChatGPT or GPT-4 to assist in software development work - generate code snippets, do some refactoring, get advice on improvements and so on. But at the end of the day LLM is merely a tool AI. It does not have it’s own judgement, values or will that human possesses. Even worse, it is debatable to what extent it knows things, as the fundamental way it operates is predicting what text should be generated next. It is the user that drives the LLM who is supposed to know what they are doing. For example, ChatGPT may generate a code snippet that is mostly correct in the light of provided requirements. It’s up to the user to make it completely correct. Now, what if the user was a non-technical entrepreneur? How well would it work out for them?

By amplifying, not replacing, human effort LLMs can be said to provide a leverage. As the leverage increases, the value of sound judgement increases as well. When it’s easier to turn your decision into a reality, the quality of these decisions starts to matter a whole lot more.

But the AI hype wave is in full swing now. There’s all kinds of social media influencers and content creators telling you to do this or that to benefit from the AI revolution to get rich. A lot of these people are digital snake oil merchants who are jumping from trend to trend. Some propose completely silly, unrealistic money making schemes just to get your attention. It’s all part of make-money-online content creation playbook. The truth is, these people mostly don’t make money from the thing they promote. Instead, they make money by promoting the thing they are promoting through ad revenue (esp. on Youtube), affiliate offers and their own info products. During the Bitcoin bull run and NFT craze there was no shortage of characters talking loudly about how you have to “invest” into this or that thingy on the blockchain. I suspect the vast majority of them could not tell the difference between Base64 and AES. Now you have great deal of people talking the same way about AI (mostly LLMs). How many of these newly minted “AI experts” even know what the fuck a linear regression is?

That is not to say that nobody ever gets rich during a gold rush. Some people swiftly grab a transient opportunity and milk it for what it’s worth. Others do some equivalent of selling shovels and pick axes during the Wild West period. Some simply gamble and happen to be winning. Speaking about LLM-based SaaS apps specifically, there’s a certain problem. If you have some SaaS app that has OpenAI API as a hard, critical dependency that means your SaaS business is essentially at the mercy of OpenAI. Right now, there are some important questions that pertain to generative AI. What exactly should and should not it be allowed to generate? If some data is scraped to train the LLM what happens to related copyrights? What will be done against gray hat and black hat usage of LLMs, such as fake product review generation, harassment, phishing, influence operations? Depending on how things play out, your entire SaaS business could die overnight by being cut off from API access as it may turn out not be kosher anymore in the light of new legislation or newly introduced corporate policy. Remember that Twitter API was free and open in the early years. There were many companion apps for Twitter users built on that API - unofficial clients, automation tools and the like. But over time Twitter introduced many restrictions into the official API and most of them had to die. This is something that indie hackers who rely on the goodwill and interest alignment from OpenAI (or Anthropic, or Google…) should keep in mind.

Furthermore, using an LLM that you don’t self-host can be problematic from data security and privacy perspective, as you are literally telling some other entity the specifics of your work. There was already a scandal of Samsung employees leaking trade secrets to OpenAI by using ChatGPT to help with their regularly scheduled programming. That is the reason why the usage of ChatGPT is explicitly banned in some corporations. A fundamental problem with information society is that many things are data driven, but we mostly don’t have any control how data about people and companies is processed, stored, transferred. Information functions by being always in motion and largely has a life of it’s own. If you tell your trade secrets to ChatGPT, can you trust ChatGPT not to tell them to your competitor?

Times are difficult now. We’re only 3.5 years into 2020s and so far we had a global pandemic, energy crisis, murderous Taliban psychos winning the Afghan war, a new major conflict in Ukraine, inflation wave causing cost of living crisis, tech downturn and banks failing. But even now I advice everyone to keep their heads cool. You probably cannot prompt your way into quick wealth, especially if you’re about average on coming up with LLM prompts. In economics there’s a concept of alpha. It’s a number quantifying how well an asset outperforms the overall market. Dumb luck aside, to achieve exceptional results that greatly outperform most of the competition one must posses or do something exceptional. The problem with doing easy things (e.g. generating basic ad copy) is that they don’t stay easy for very long in a competitive environment. It is unlikely that ChatGPT, GPT-4 and other AI systems will make everyone jobless due to considerations I have discussed. These systems are merely tools subservient to a human will and there will be a need for people working with them. Perhaps the daily reality of many occupations will change, perhaps some people will need to learn how to do something else, but I don’t expect the fully AI-driven world to happen anytime soon.

We cannot know what the future will be like. In a sense, we don’t even have a (predictable) future when things are so crazy, volatile and fucked up. All we can do is hedge our bets and roll the dice.

Keep calm and seek your alpha.

Understanding JavaScript packers

Tue, 30 May 2023 00:00:00 +0000

In the world of desktop software, the concept of packer is not new. A packer is a tool that takes a binary executable file as input, applies transformations (e.g. compression, encryption, introducing anti-debugging tricks) and outputs a new, modified executable file that is different at binary level, but retains the functionality of original program. Some packers, such as UPX are only meant to make the executable files smaller. More advanced packers apply machine code encryption to make reverse engineering harder. The key to decrypt the machine code is hidden in a binary file and applied automatically when program is launched, but the encryption layer introduces a barrier for disassebling or decompiling the machine code, which increases the effort needed to reverse engineer the inner workings of a program.

But that’s old news. Nowadays a lot of software is web-based. But the thing is, the concept of packer is also applicable in JavaScript world. There are tools and techniques to transform JS code in such a way that it becomes smaller and/or less readable. Thus JS packing can be used as obfuscation technique. Note that I am not talking about simple code minification here. Instead, we’re going to see some techniques that are similar to executable packing from the compiled software world.

Depending on what kind of hat the programmer is wearing, there might be different motivations to apply JS packing. One is simply to make JS code smaller, which helps performance. Lighter pages load faster, which leads to better user experience and may have a positive impact on search engine ranking. JS packing can also be applied to make things harder for web scraper developers, esp. if there’s some non-trivial client side functionality involved. For instance, Imperva (formely known as Incapsula) antibot solution is known to have used a simple JS packing technique as one layer of JS SDK code obfuscation. On the black hat side, JS packing is used to evade detection of malicious code that is placed on compromised or fraudulent websites for cybercrime purposes.

There are two aspects to JavaScript packing:

Transforming the original code into a new (e.g. compressed, encoded, encrypted) form. This will be hardcoded into output code.
Introducing a small amount of helper code that will unpack and execute the original code.

eval(unescape()) is the minimum viable JS packing technique. Original code is converted into URL-encoded form, then put as an argument to eval(unescape()) functions calls.

To find an example, we can search for "eval(unescape(" on PublicWWW. On one of the pages we find, there’s a following JS snippet:

var myText = 'Пишите нам'; eval(unescape('%64%6f%63%75%6d%65%6e%74%2e%77%72%69%74%65%28%27%3c%61%20%68%72%65%66%3d%22%6d%61%69%6c%74%6f%3a%76%69%74%40%61%75%64%69%74%2d%69%74%2e%72%75%3f%73%75%62%6a%65%63%74%3d%25%44%30%25%41%31%25%44%30%25%42%31%25%44%30%25%42%35%25%44%31%25%38%30%25%44%30%25%42%45%25%44%30%25%42%43%25%44%30%25%42%35%25%44%31%25%38%32%25%44%31%25%38%30%22%3e%27%2b%28%6d%79%54%65%78%74%20%3f%20%6d%79%54%65%78%74%20%3a%20%27%76%69%74%40%61%75%64%69%74%2d%69%74%2e%72%75%27%29%2b%27%3c%2f%61%3e%27%29%3b'))

Screenshot 1

We can trivially recover the original source code of this snippet by copy-pasting the unescape() call into JS REPL:

$ node
Welcome to Node.js v20.1.0.
Type ".help" for more information.
> unescape('%64%6f%63%75%6d%65%6e%74%2e%77%72%69%74%65%28%27%3c%61%20%68%72%65%66%3d%22%6d%61%69%6c%74%6f%3a%76%69%74%40%61%75%64%69%74%2d%69%74%2e%72%75%3f%73%75%62%6a%65%63%74%3d%25%44%30%25%41%31%25%44%30%25%42%31%25%44%30%25%42%35%25%44%31%25%38%30%25%44%30%25%42%45%25%44%30%25%42%43%25%44%30%25%42%35%25%44%31%25%38%32%25%44%31%25%38%30%22%3e%27%2b%28%6d%79%54%65%78%74%20%3f%20%6d%79%54%65%78%74%20%3a%20%27%76%69%74%40%61%75%64%69%74%2d%69%74%2e%72%75%27%29%2b%27%3c%2f%61%3e%27%29%3b')
`document.write('<a href="mailto:[REDACTED]?subject=%D0%A1%D0%B1%D0%B5%D1%80%D0%BE%D0%BC%D0%B5%D1%82%D1%80">'+(myText ? myText : '[REDACTED]')+'</a>');`

This turned out to be a simple trick to hide owner’s email address from web scrapers that don’t execute JavaScript and are not tailored to this specific site.

A more complex example would be Dean Edwards packer. Let us pack the following JavaScript snippet:

let hello = "hello world";
console.log(hello);

After checking both checkboxes and pressing “Pack” button we get the following packed code:

eval(function(p,a,c,k,e,r){e=String;if(!''.replace(/^/,String)){while(c--)r[c]=k[c]||c;k=[function(e){return r[e]}];e=function(){return'\\w+'};c=1};while(c--)if(k[c])p=p.replace(new RegExp('\\b'+e(c)+'\\b','g'),k[c]);return p}('1 0="0 2";3.4(0);',5,5,'hello|let|world|console|log'.split('|'),0,{}))

Using Beautifier.io to prettify (but not unpack at the moment - make sure all checkboxes are unchecked) this code makes it slightly more readable:

eval(function(p, a, c, k, e, r) {
    e = String;
    if(!''.replace(/^/, String)) {
        while(c--) r[c] = k[c] || c;
        k = [function(e) {
            return r[e]
        }];
        e = function() {
            return '\\w+'
        };
        c = 1
    };
    while(c--)
        if(k[c]) p = p.replace(new RegExp('\\b' + e(c) + '\\b', 'g'), k[c]);
    return p
}('1 0="0 2";3.4(0);', 5, 5, 'hello|let|world|console|log'.split('|'), 0, {}))

Screenshot 2

In the outermost layer we see that eval() function takes a JS code string returned by immediately invoked function with five parameters - p, a, c, k, e and r. The values for these arguments are related to original code snippet and in aggregate represent a packed representation. The innermost function applies some regular expression magic to regenerate what the code was originally, so that eval() could execute it.

Like with the previous example, we can skip the eval() part and perform only the unpacking:

> let unpack = function(p,a,c,k,e,r){e=String;if(!''.replace(/^/,String)){while(c--)r[c]=k[c]||c;k=[function(e){return r[e]}];e=function(){return'\\w+'};c=1};while(c--)if(k[c])p=p.replace(new RegExp('\\b'+e(c)+'\\b','g'),k[c]);return p};
undefined
> unpack('1 0="0 2";3.4(0);',5,5,'hello|let|world|console|log'.split('|'),0,{});
'let hello="hello world";console.log(hello);'

We got the original code without newline in the middle. Like with other obfuscation techniques, we are not necessarily able to recover the exact original code when reverse engineering.

A lot of the online JS packer tools are some variations of Dean Edwards packer that was originally developed in C#. Thus it is useful to know how to revert what it does to the code.

Scraping product data from Nike.com

Mon, 15 May 2023 00:00:00 +0000

Suppose you are looking to collect pricing data on male footwear from the official website of one of the industry leaders - Nike.com. There’s a product list page that seems like a good place to start, but the infinite scroll feature might seem puzzling to budding web scraper developers. In this post, we will go through developing a Scrapy project for scraping sneaker price data from this website. Some familiarity with Scrapy and web scraping in general is assumed.

Before we proceed with writing the code, let us explore the site a little. Open the source code of product list page and try searching for a product title that we can see rendered on the page. The purpose of doing this is to gain insight of how data is represented in the original HTML that is fetched on the first load of the page. By searching for “Nike Air Max Pulse” we get some hits with this string being part of normal HTML, but more importantly we get a hit in the big JSON document that is embedded in the page. We can see that data on products (titles, prices, image URLs) is encoded into this JSON string as well.

Screenshot 1

By scrolling up, we find that the JSON string is wrapped in a script element (due to frontend programming reasons):

<script id="__NEXT_DATA__" type="application/json">

Screenshot 2

This is good for us, as we can extract the data we need from big JSON string and use it for our purposes. However, how does the infinite scroll work? How is the new data loaded? This can be explored in Network tab of Chrome DevTools.

Screenshot 3 Screenshot 4

We can see that site launched an API call to api.nike.com every time new page of product list data is needed (i.e. you scroll to the bottom and trigger the fetching of new data). The JSON response to this API request contains product data (largely in the same structure as we found in the page source) and a section for pagination that contains some relative URLs (starting with /product_feed/rollup_threads/v2) that are passed into endpoint URL parameter when further pages of data are being loaded. The initial value for endpoint parameter that matches second page of data can also be found in the big JSON in the page source:

"pageData":{"prev":"","next":"/product_feed/rollup_threads/v2?filter=marketplace%28US%29\u0026filter=language%28en%29\u0026filter=employeePrice%28true%29\u0026filter=attributeIds%2816633190-45e5-4830-a068-232ac7aea82c%2C0f64ecc7-d624-4e91-b171-b83a03dd8550%29\u0026anchor=24\u0026consumerChannelId=d9a5bc42-4b9c-4976-858a-f159cf99c647&count=24

Screenshot 5

This informs our general strategy for scraping:

Fetch the product list page HTML with nested JSON. Use it to extract some initial product data and the first value for endpoint parameter to start API scraping.
Scrape product feed API by traversing all pages. This gives product titles and pricing info.
Augment data from step 2 by scraping product detail pages to add some more data on products.

Step 3 is technically not needed if we only want pricing information, but I’m including it here as many web scraping project for ecommerce industry involve scraping the PDP (product detail page) as well.

After new project boilerplate is generated with scrapy startproject and genspider commands let us start by editing items.py to define a class for an item (a unit of scraped data):

# Define here the models for your scraped items
#
# See documentation in:
# https://docs.scrapy.org/en/latest/topics/items.html

import scrapy


class NikeProductItem(scrapy.Item):
    title = scrapy.Field()
    subtitle = scrapy.Field()
    pid = scrapy.Field()
    current_price = scrapy.Field()
    empl_price = scrapy.Field()
    full_price = scrapy.Field()
    in_stock = scrapy.Field()
    product_url = scrapy.Field()
    image_urls = scrapy.Field()
    description = scrapy.Field()

The Scrapy spider class I generated is named NikecomSpider and resides in nikecom.py file. Let’s make this spider load the product list page and extract some initial product data:

class NikecomSpider(scrapy.Spider):
    name = 'nikecom'
    allowed_domains = ['nike.com', 'www.nike.com', 'api.nike.com']
    start_urls = ['https://www.nike.com/w/mens-shoes-nik1zy7ok']

    def start_requests(self):
        for url in self.start_urls:
            yield scrapy.Request(url, callback=self.parse_first_page)

    def convert_product_dict_into_item(self, product):
        # Removed for brevity.

    def parse_first_page(self, response):
        next_data_json_str = response.xpath('//script[@id="__NEXT_DATA__"]/text()').get()

        json_dict = json.loads(next_data_json_str)
        props = json_dict.get("props")
        page_props = props.get("pageProps")
        initial_state = page_props.get("initialState")
        wall = initial_state.get("Wall")
        products = wall.get('products')

        for product in products:
            item = self.convert_product_dict_into_item(product)
            yield item
            # TODO: yield request to product page

I have created method convert_product_dict_into_item() that converts data from original JSON to Scrapy item object, but for now this code is not shown for the sake of brevity. Just keep in mind that it will be reused when we do API scraping.

Next thing we want to do in parse_first_page() method is to launch the first API request. We do this based on initial endpoint parameter value that we extract from JSON string:

        page_data = wall.get('pageData')
        if page_data is not None:
            next_page_link = page_data.get("next")
            if next_page_link is not None and next_page_link != '':
                yield self.create_feed_api_request(next_page_link)

Here we rely on another helper method to create the API request:

    def create_feed_api_request(self, next_page_link):
        params = {
            'queryid': 'products',
            'anonymousId': '7CC266B713D36CCC7275B33B6E4F9206', #XXX
            'country': 'us',
            'endpoint': next_page_link,
            'language': 'en',
            'localizedRangeStr': '{lowestPrice} — {highestPrice}'
        }

        url = 'https://api.nike.com/cic/browse/v2'

        url = url + '?' + urlencode(params)

        return scrapy.Request(url, callback=self.parse_product_feed)

In parse_product_feed() method we process the API response, create item objects and generate requests to product detail pages:

    def parse_product_feed(self, response):
        json_str = response.text

        json_dict = json.loads(json_str)

        products = json_dict.get('data', dict()).get('products', dict()).get('products', [])

        for product in products:
            item = self.convert_product_dict_into_item(product)
            yield scrapy.Request(item['product_url'], meta={'item': item}, callback=self.parse_product_page)

        next_page_link = json_dict.get('data', dict()).get('products', dict()).get('pages', dict()).get('next')
        if next_page_link is not None and next_page_link != '':
            yield self.create_feed_api_request(next_page_link)

Note that partially-filled item object is passed to the next callback via the meta dictionary. We modify the parse_first_page() method to do this as well, instead of just generating item object.

In this sample project product detail page parsing is limited to extracting only a couple of fields from the HTML:

    def parse_product_page(self, response):
        item = response.meta.get('item')
        
        item['description'] = response.xpath('//div[contains(@class, "description-preview")]/p/text()').get()
        item['image_urls'] = response.xpath('//picture/source/@srcset').getall()

        yield item

This now generates the final item object, with product description and image URLs:

{'current_price': 130,
 'description': 'Nothing as fly, nothing as comfortable, nothing as proven. '
                'The Nike Air Max 90 stays true to its OG running roots with '
                'the iconic Waffle sole, stitched overlays and classic TPU '
                'details. Classic colors celebrate your fresh look while Max '
                'Air cushioning adds comfort to the journey.',
 'empl_price': 78,
 'full_price': 130,
 'image_urls': ['https://static.nike.com/a/images/t_PDP_1280_v1/f_auto,b_rgb:f5f5f5/wzitsrb4oucx9jukxsmc/air-max-90-mens-shoes-6n3vKB.png',
                'https://static.nike.com/a/images/t_PDP_1280_v1/f_auto,b_rgb:f5f5f5/wzitsrb4oucx9jukxsmc/air-max-90-mens-shoes-6n3vKB.png',
                'https://static.nike.com/a/images/f_auto,b_rgb:f5f5f5,w_640/wzitsrb4oucx9jukxsmc/air-max-90-mens-shoes-6n3vKB.png',
                'https://static.nike.com/a/images/t_PDP_864_v1/f_auto,b_rgb:f5f5f5/wzitsrb4oucx9jukxsmc/air-max-90-mens-shoes-6n3vKB.png',
                'https://static.nike.com/a/images/t_PDP_864_v1/f_auto,b_rgb:f5f5f5/wzitsrb4oucx9jukxsmc/air-max-90-mens-shoes-6n3vKB.png',
                'https://static.nike.com/a/images/f_auto,b_rgb:f5f5f5,w_440/wzitsrb4oucx9jukxsmc/air-max-90-mens-shoes-6n3vKB.png',
                'https://static.nike.com/a/images/t_PDP_864_v1/f_auto,b_rgb:f5f5f5/wzitsrb4oucx9jukxsmc/air-max-90-mens-shoes-6n3vKB.png',
                'https://static.nike.com/a/images/t_PDP_864_v1/f_auto,b_rgb:f5f5f5/wzitsrb4oucx9jukxsmc/air-max-90-mens-shoes-6n3vKB.png',
                'https://static.nike.com/a/images/f_auto,b_rgb:f5f5f5,w_440/wzitsrb4oucx9jukxsmc/air-max-90-mens-shoes-6n3vKB.png',
                'https://static.nike.com/a/images/t_PDP_1280_v1/f_auto,b_rgb:f5f5f5/wzitsrb4oucx9jukxsmc/air-max-90-mens-shoes-6n3vKB.png',
                'https://static.nike.com/a/images/t_PDP_1280_v1/f_auto,b_rgb:f5f5f5/wzitsrb4oucx9jukxsmc/air-max-90-mens-shoes-6n3vKB.png',
                'https://static.nike.com/a/images/f_auto,b_rgb:f5f5f5,w_640/wzitsrb4oucx9jukxsmc/air-max-90-mens-shoes-6n3vKB.png',
                'https://static.nike.com/a/images/t_PDP_864_v1/f_auto,b_rgb:f5f5f5/wzitsrb4oucx9jukxsmc/air-max-90-mens-shoes-6n3vKB.png',
                'https://static.nike.com/a/images/t_PDP_864_v1/f_auto,b_rgb:f5f5f5/wzitsrb4oucx9jukxsmc/air-max-90-mens-shoes-6n3vKB.png',
                'https://static.nike.com/a/images/f_auto,b_rgb:f5f5f5,w_440/wzitsrb4oucx9jukxsmc/air-max-90-mens-shoes-6n3vKB.png',
                'https://static.nike.com/a/images/t_PDP_864_v1/f_auto,b_rgb:f5f5f5/wzitsrb4oucx9jukxsmc/air-max-90-mens-shoes-6n3vKB.png',
                'https://static.nike.com/a/images/t_PDP_864_v1/f_auto,b_rgb:f5f5f5/wzitsrb4oucx9jukxsmc/air-max-90-mens-shoes-6n3vKB.png',
                'https://static.nike.com/a/images/f_auto,b_rgb:f5f5f5,w_440/wzitsrb4oucx9jukxsmc/air-max-90-mens-shoes-6n3vKB.png'],
 'in_stock': True,
 'pid': '12788919',
 'product_url': 'https://www.nike.com/t/air-max-90-mens-shoes-6n3vKB/CN8490-002',
 'subtitle': "Men's Shoes",
 'title': 'Nike Air Max 90'}

The final Scrapy spider code is as follows:

import scrapy

import json
from urllib.parse import urlencode

from nike.items import NikeProductItem

class NikecomSpider(scrapy.Spider):
    name = 'nikecom'
    allowed_domains = ['nike.com', 'www.nike.com', 'api.nike.com']
    start_urls = ['https://www.nike.com/w/mens-shoes-nik1zy7ok']

    def start_requests(self):
        for url in self.start_urls:
            yield scrapy.Request(url, callback=self.parse_first_page)

    def convert_product_dict_into_item(self, product):
        item = NikeProductItem()
        
        item['title'] = product.get("title")
        item['subtitle'] = product.get("subtitle")
        item['pid'] = product.get("pid")
        
        price_dict = product.get("price", dict())
        if price_dict is not None:
            item['current_price'] = price_dict.get("currentPrice")
            item['empl_price'] = price_dict.get("employeePrice")
            item['full_price'] = price_dict.get("fullPrice")

        item['in_stock'] = product.get("inStock")
        item['product_url'] = product.get('url')
        if item['product_url'] is not None:
            item['product_url'] = item['product_url'].replace("{countryLang}", "https://www.nike.com")

        return item

    def create_feed_api_request(self, next_page_link):
        params = {
            'queryid': 'products',
            'anonymousId': '7CC266B713D36CCC7275B33B6E4F9206', #XXX
            'country': 'us',
            'endpoint': next_page_link,
            'language': 'en',
            'localizedRangeStr': '{lowestPrice} — {highestPrice}'
        }

        url = 'https://api.nike.com/cic/browse/v2'

        url = url + '?' + urlencode(params)

        return scrapy.Request(url, callback=self.parse_product_feed)

    def parse_first_page(self, response):
        next_data_json_str = response.xpath('//script[@id="__NEXT_DATA__"]/text()').get()

        json_dict = json.loads(next_data_json_str)
        props = json_dict.get("props")
        page_props = props.get("pageProps")
        initial_state = page_props.get("initialState")
        wall = initial_state.get("Wall")
        products = wall.get('products')

        for product in products:
            item = self.convert_product_dict_into_item(product)
            if item.get('product_url') is not None:
                yield scrapy.Request(item['product_url'], meta={'item': item}, callback=self.parse_product_page)

        page_data = wall.get('pageData')
        if page_data is not None:
            next_page_link = page_data.get("next")
            if next_page_link is not None and next_page_link != '':
                yield self.create_feed_api_request(next_page_link)

    def parse_product_page(self, response):
        item = response.meta.get('item')
        
        item['description'] = response.xpath('//div[contains(@class, "description-preview")]/p/text()').get()
        item['image_urls'] = response.xpath('//picture/source/@srcset').getall()

        yield item

    def parse_product_feed(self, response):
        json_str = response.text

        json_dict = json.loads(json_str)

        products = json_dict.get('data', dict()).get('products', dict()).get('products', [])

        for product in products:
            item = self.convert_product_dict_into_item(product)
            yield scrapy.Request(item['product_url'], meta={'item': item}, callback=self.parse_product_page)

        next_page_link = json_dict.get('data', dict()).get('products', dict()).get('pages', dict()).get('next')
        if next_page_link is not None and next_page_link != '':
            yield self.create_feed_api_request(next_page_link)

Running this requires no proxies or anything to deal with antibots despite Nike being a customer of not one, but two antibot solutions - Kasada and Akamai Bot Manager. This provides an interesting example of how there are multiple levels to protection against automation. Automatically creating Nike accounts and/or placing orders is known to be challenging thing to do, yet Nike allows a simple form of product data scraping without imposing any difficulty on doing that. What we developed here was a quite simple Scrapy project that was based on some quite basic forms of reverse engineering.

One last thing to note is the discounted employee price that we were able to scrape from API. This is a slightly hidden piece of information that regular web user is not supposed to see, but one can scrape it by knowing how to tap into the API.

How do they (try to) block bots: a broad review of automation countermeasures

Sun, 14 May 2023 00:00:00 +0000

Not every website wants to let the data to be scraped and not every app wants to allow automation of user activity. If you work in scraping and automation at any capacity you certainly have dealt with sites that work just fine when accessed through normal browser throwing captchas or error pages at your bot. There are multiple security mechanisms that can cause this to happen. Today we will do a broad review of automation countermeasures that can be implemented at various levels.

IP level

The following techniques work at Internet Protocol level by allowing or blocking traffic based on source IP address it is coming from.

Geofencing

Geofencing (or geo-restricting/geo-blocking) is blocking/allowing requests based on source geographic location (typically a country). This relies on GeoIP data provided by vendors such as MaxMind or IP2Location to perform lookups.

Screenshot 1

As web scraper developer you can trivially bypass this using proxies.

Rate limiting

Web sites may impose upper bounds on on how many requests are allowed per some timeframe from a single IP and start blocking incoming traffic if traffic intensity exceeds that threshold. This is typically implemented with a leaky bucket algorithm. Suppose the site allows 60 requests per minute from a single IP (i.e. one per seconds). It keeps a counter of how many requests it received, decreasing it by one every second. If the counter exceeds the threshold value (60) the site refuses further requests until it is below the threshold again. This limits how aggressive we can be when scraping, but typically can be defeated by spreading the traffic around a set of IP addresses by routing it through proxy pool.

Filtering by AS/ISP

This can be seen as a variation of geofencing, but blocking can also be performed based on source ISP or Autonomous System by performing Autonomous System Lookups. For example, sites or antibot vendors may explicitly disallow traffic that comes from data centers or major cloud vendors (AWS, Digital Ocean, Azure and so on).

Filtering by IP reputation

Proxies can be used to bypass the above ways to block traffic, but blocking can also be performed based on IP address reputation data from threat intelligence vendors like IPQualityScore. So if your proxy provider is a sketchy one that is built on foundation of botnet it’s proxies may be listed in some blacklist, which makes the trust score bad. One must take care to use a reputable proxy provider.

HTTPS level

The following techniques work at HTTP and TLS protocol levels.

HTTP headers and cookies

The very simplest way to know that traffic is automated is to check User-Agent header in HTTP request. For example, curl puts something like curl/8.0.1 in there and some headless browsers actually let the server to know they are running in headless mode (unless configured otherwise). But that’s trivial to defeat by just reproducing the header value programmatically. One baby step further is to not only check the User-Agent header, but also other request headers to make sure like the ones in the browser in aggregate. This is also easy to defeat by reproducing all the headers in a request. In fact, Chrome DevTools Network tab has a handy feature of letting you conveniently copy a request as a curl command or JavaScript (or even PowerShell!) snippet.

Screenshot 2

Cookies also matter when it comes to dealing with sites that are hostile to automation, as cookieless requests might get rejected. Depending on exact specifics of the site, getting the cookies might be as easy as just loading the front page with requests.Session or it might involve more elaborate trickery if antibot vendor or captcha service is issuing them based on browser assesment and user activity (more on that later).

HTTP/2 fingerprinting

We covered the simple stuff, now lets get to something more complicated - HTTP/2 fingerprinting. HTTP/2 is quite complex binary protocol that leaves quite a bit of wiggle room to implementor. The way it is implemented in curl is different than implementation in Google Chrome (altough both are RFC-compliant). Traffic generated by each implementation can reliably betray what kind of implementation it is to the server, CDN or WAF. I have covered this in greated detail in another post earlier.

TLS fingerprinting

HTTP/2 is pretty much never used in plaintext form. After all, you cannot let the dirty people from NSA to just watch the unencrypted traffic without any problems. However, TLS protocol is also very complex. It involves a great deal of technical details in the client traffic (protocol version, list of extensions, proposed cryptographic protocols, etc.) that betray what kind of client software it comes from. In fact, a single ClientHello message can be used to quite reliably predict the OS and browser that generated it. There’s also post earlier in this blog to cover this topic in greater detail.

Application level

The following techniques deal with probing the client-side environment (typically a web browser) to detect and/or prevent automation.

JS environment checking

Modern web browsers are extremely complex software systems. They’re not just user-facing applications to load some pages from remote servers anymore. As of 2023, Chrome or Firefox contains an entire programmable environment for client side software development that is primarily done in JS or variants thereof (that end up being transpiled into JS anyway). Naturally this means that there’s quite a bit of stuff in the JavaScript environment that sites can check for tell-tale signs of browser being controlled programmatically by automation software like like Selenium, Playwright or Puppetteer.

Some examples:

navigator.webdriver=true generally betrays browser automation.
Presence of document.$cdc_asdjflasutopfhvcZLmcfl_ object betrays Selenium.
On Chromium-based browsers, window.chrome object not being available means the browser is running in headless mode.

Browser and device fingerprinting

Other than aforementioned tell-tale signs of automation, client side JS code can gather a great deal of information on the system it is running on: installed fonts, screen resolution, supported browser features, hardware components and so on. Each of them is a trait of hardware-software setup the code is running on. Furthermore, exact rendering of graphics (through HTML canvas or WebGL API) is dependent on hardware/software setup, i.e. Chrome on macOS/Apple Silicon will not render the exact same image as Firefox on Linux/x86 - some pixels will be slightly, but consistently off, which is another piece of information that can be kept track of. A collection of these traits is referred to as fingerprint.

Fingerprinting is part of anti-automation toolkit. For example, social media apps are able to detect when login session is launched from a new devide vs. the same device being used for new user session. Furthermore, it can be quite telling when a cluster of accounts share the same fingerprint (in the case of e.g. device farm or fully virtualised botting operation).

Javascript challenges

To make requests-based bots harder to implement, site may want to check if client is capable of running Javascript before giving the data. For example, Cloudflare has an Email Address Obfuscation feature that relies on client-side JS code running in the browser to convert the email address from encoded form in the HTML document your browser gets from CDN to the normal form. Notice the difference between what you see in Elements tab of Chrome DevTools and in the original page source for the page of site using this feature.

Screenshot 3 Screenshot 4

For this particular example of data obfuscation, there’s a Stack Overflow answer providing a small Python snippet to decode it back. However, if you are doing requests-based scraping and rely on regular expressions to extract email addresses this sort of stuff can already cause at least some difficulty.

Furthermore, JS challenges can be more elaborate than this. Depending on how Cloudflare’s antibot features are configured browser may be required to run some JS code with cryptographic proof of work stuff, provide the result of the computation to CF and only then get the cookie that allows it to properly access anything on the site.

User level

The following techniques involve behavioural tracking of user activity, checking that the user is actually human and leveraging user identitities (accounts) to fight automation.

CAPTCHA

A captcha is often (but not always) an incovenient silly puzzle that you have to solve to prove you’re not a robot by picking the segments containing certain object or something like that. That’s a visible captcha. Some captchas (e.g. reCaptcha v3) are invisible - they work in background tracking user activity (mouse movement, etc.) to assess if it looks human or robotic. The puzzle UI is displayed only when there is a suspicion that user is a bot.

The underlying idea for this is that certain tasks are difficult to do programmatically, but easy to do for a person.

Account-level throttling

A simple way to hinder automation is to rate-limit user activity per account. This is commonly done by social media platforms. If user activity exceeds the rate limits the site/app can simply reject the requests or even ban the account.

Account banning

No suprise here - ban hammer is part of anti-automation arsenal as well. Misbehaving accounts can be banned. This is of particular importance to social media automation - an account ban can mean a setback or even outright failure of social media growth hacking efforts.

Making account creation difficult

Banning and throttling user accounts would not help much if it’s possible to create a lot of them quickly. Thus making account creation difficult to automate (by requiring phone verification, integrating third party antibot solutions, making request flows hard to reproduce) is one more thing that sites/apps can do to fight automation.

Realistically, it is near impossible to completely prevent botting if there’s an a sufficient incentive for people to engage in automation. However, the economic feasibility of automation can be weakened by putting barriers that make it more expensive such as phone or ID verification for new accounts and rate-limiting/restricting/banning existing accounts for sketchy, anomalous activity.

Mouse activity monitoring

Client-side JS code can monitor mouse movements and submit them to some neural network running on the backend to assess if they look human or not. This is performed not only by some captcha solutions, but also antibot vendors as well. For example, Akamai Bot Manager phones back a sample of user’s mouse movement as part of clearance cookie issuance flow.

CrawlSpider and link extractors for rule-based crawling in Scrapy

Tue, 02 May 2023 00:00:00 +0000

The front page of Scrapy project provides a basic example of Scrapy spider:

class BlogSpider(scrapy.Spider):
    name = 'blogspider'
    start_urls = ['https://www.zyte.com/blog/']

    def parse(self, response):
        for title in response.css('.oxy-post-title'):
            yield {'title': title.css('::text').get()}

        for next_page in response.css('a.next'):
            yield response.follow(next_page, self.parse)

In this code snippet, crawling is performed imperatively - link to next page is extracted and from it a new request is generated. However, Scrapy also supports another, declarative, approach of crawling that is based on setting crawling rules for the spider and letting it follow links without explicit request generation. This abstracts away the link traversal and enables us to focus on data extraction aspect.

To use this approach in our code, we must be familiar with LinkExtractor objects and CrawlSpider class. Let us start with the former. For the sake of the example, we will be doing a little scraping of books.toscrape.com - a simple site that is made to be scraped.

A link extractor is an object that processes a Scrapy Response object and extracts a list of links according to some criteria (URL substrings, regexes, XPaths, CSS selectors, subdomains). It returns the extracted links as a list of Link objects. Each Link object contains not only the absolute URL, but also some additional information: anchor text, URL fragment, nofollow flag.

Let us use Scrapy shell to see how link extractors can be created and used:

In [1]: fetch("https://books.toscrape.com/")
2023-05-02 13:05:41 [scrapy.core.engine] INFO: Spider opened
2023-05-02 13:05:42 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://books.toscrape.com/> (referer: None)

In [2]: from scrapy.linkextractors import LinkExtractor

In [3]: book_link_extractor = LinkExtractor(allow='index.html', deny=['catalogue/page', 'catalogue/category/books', 'https://books.toscrape.com/index.html'])

In [4]: for link in book_link_extractor.extract_links(response):
   ...:     print(link)
   ...: 
Link(url='https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html', text='', fragment='', nofollow=False)
Link(url='https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html', text='', fragment='', nofollow=False)
Link(url='https://books.toscrape.com/catalogue/soumission_998/index.html', text='', fragment='', nofollow=False)
Link(url='https://books.toscrape.com/catalogue/sharp-objects_997/index.html', text='', fragment='', nofollow=False)
Link(url='https://books.toscrape.com/catalogue/sapiens-a-brief-history-of-humankind_996/index.html', text='', fragment='', nofollow=False)
Link(url='https://books.toscrape.com/catalogue/the-requiem-red_995/index.html', text='', fragment='', nofollow=False)
Link(url='https://books.toscrape.com/catalogue/the-dirty-little-secrets-of-getting-your-dream-job_994/index.html', text='', fragment='', nofollow=False)
Link(url='https://books.toscrape.com/catalogue/the-coming-woman-a-novel-based-on-the-life-of-the-infamous-feminist-victoria-woodhull_993/index.html', text='', fragment='', nofollow=False)
Link(url='https://books.toscrape.com/catalogue/the-boys-in-the-boat-nine-americans-and-their-epic-quest-for-gold-at-the-1936-berlin-olympics_992/index.html', text='', fragment='', nofollow=False)
Link(url='https://books.toscrape.com/catalogue/the-black-maria_991/index.html', text='', fragment='', nofollow=False)
Link(url='https://books.toscrape.com/catalogue/starving-hearts-triangular-trade-trilogy-1_990/index.html', text='', fragment='', nofollow=False)
Link(url='https://books.toscrape.com/catalogue/shakespeares-sonnets_989/index.html', text='', fragment='', nofollow=False)
Link(url='https://books.toscrape.com/catalogue/set-me-free_988/index.html', text='', fragment='', nofollow=False)
Link(url='https://books.toscrape.com/catalogue/scott-pilgrims-precious-little-life-scott-pilgrim-1_987/index.html', text='', fragment='', nofollow=False)
Link(url='https://books.toscrape.com/catalogue/rip-it-up-and-start-again_986/index.html', text='', fragment='', nofollow=False)
Link(url='https://books.toscrape.com/catalogue/our-band-could-be-your-life-scenes-from-the-american-indie-underground-1981-1991_985/index.html', text='', fragment='', nofollow=False)
Link(url='https://books.toscrape.com/catalogue/olio_984/index.html', text='', fragment='', nofollow=False)
Link(url='https://books.toscrape.com/catalogue/mesaerion-the-best-science-fiction-stories-1800-1849_983/index.html', text='', fragment='', nofollow=False)
Link(url='https://books.toscrape.com/catalogue/libertarianism-for-beginners_982/index.html', text='', fragment='', nofollow=False)
Link(url='https://books.toscrape.com/catalogue/its-only-the-himalayas_981/index.html', text='', fragment='', nofollow=False)

We fetched the front page of the site, imported LinkExtractor class and initialised an extractor for links pointing to what eCommerce people would call Product Detail Pages (PDPs). The allow and deny values of LinkExtractor constructor arguments are written so that they narrow down the exact links we need. Note that we can allow or deny links based on a single substring or a list of them. Once we have the link extractor created, we call it’s extract_links() method on a response object we got from fetching the page, which gives us a list of Link objects. We can see that all the links point to book details pages.

But we also need a link to the next Product List Page. Extracting it is also quite simple with another link extractor:

In [5]: next_page_link_extractor = LinkExtractor(allow='catalogue/page')

In [6]: next_page_link_extractor.extract_links(response)
Out[6]: [Link(url='https://books.toscrape.com/catalogue/page-2.html', text='next', fragment='', nofollow=False)]

Now that we are familiar with link extractors, let us learn about the CrawlSpider. Let’s use the Scrapy CLI to generate all the boilerplate for Scrapy project and a new spider class from the crawl template:

$ scrapy startproject sample .
$ scrapy genspider -t crawl books books.toscrape.com

In sample/spiders/books.py file we have the following starter code:

import scrapy
from scrapy.linkextractors import LinkExtractor
from scrapy.spiders import CrawlSpider, Rule


class BooksSpider(CrawlSpider):
    name = "books"
    allowed_domains = ["books.toscrape.com"]
    start_urls = ["http://books.toscrape.com/"]

    rules = (Rule(LinkExtractor(allow=r"Items/"), callback="parse_item", follow=True),)

    def parse_item(self, response):
        item = {}
        #item["domain_id"] = response.xpath('//input[@id="sid"]/@value').get()
        #item["name"] = response.xpath('//div[@id="name"]').get()
        #item["description"] = response.xpath('//div[@id="description"]').get()
        return item

This is a bit different than the default starter code you get from the default template. BooksSpider class is inheriting from CrawlSpider class (which is a subclass of scrapy.Spider). It has a rules property with a tuple of Rule objects (just a single entry at this point) and no parse() method, as the parse() method is overriden in the CrawlSpider class. We also got one callback method that is referenced in the rule by name.

Let us use this code as starting point to try out the rule-based crawling on our target site:

import scrapy
from scrapy.linkextractors import LinkExtractor
from scrapy.spiders import CrawlSpider, Rule


class BooksSpider(CrawlSpider):
    name = "books"
    allowed_domains = ["books.toscrape.com"]
    start_urls = ["http://books.toscrape.com/"]

    rules = (
        Rule(LinkExtractor(allow='index.html', 
                           deny=[
                               'catalogue/page', 
                               'catalogue/category/books', 
                               'https://books.toscrape.com/index.html']
                           ), callback='parse_item', follow=True),
        Rule(LinkExtractor(allow='catalogue/page'), follow=True)
    )


    def parse_item(self, response):
        item = {}

        item['title'] = response.xpath('//h1/text()').get()
        item['price'] = response.xpath('//div[contains(@class, "product_main")]/p[@class="price_color"]/text()').get()
        item['url'] = response.url
        
        return item

In the rules tuple we got two Rule objects that are based on link extractors we tried out earlier. The first rule is based on PDP link extractor and references the parse_item() callback method. The second one is merely for traversing to next page of the book list and is not directly associated with data extraction. Thus it references no callback method. Since the objective here is to demonstrate rule-based crawling we only extract a couple of fields from the book details page.

Running this spider scrapes the entire site without us having to explicitly generate requests to progress the crawling further. We see log lines like:

2023-05-02 13:23:20 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://books.toscrape.com/catalogue/page-2.html> (referer: http://books.toscrape.com/)
2023-05-02 13:23:20 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://books.toscrape.com/catalogue/its-only-the-himalayas_981/index.html> (referer: http://books.toscrape.com/)
2023-05-02 13:23:20 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://books.toscrape.com/catalogue/libertarianism-for-beginners_982/index.html> (referer: http://books.toscrape.com/)
2023-05-02 13:23:20 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://books.toscrape.com/catalogue/mesaerion-the-best-science-fiction-stories-1800-1849_983/index.html> (referer: http://books.toscrape.com/)
2023-05-02 13:23:20 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://books.toscrape.com/catalogue/olio_984/index.html> (referer: http://books.toscrape.com/)
2023-05-02 13:23:20 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://books.toscrape.com/catalogue/our-band-could-be-your-life-scenes-from-the-american-indie-underground-1981-1991_985/index.html> (referer: http://books.toscrape.com/)
2023-05-02 13:23:20 [scrapy.core.scraper] DEBUG: Scraped from <200 http://books.toscrape.com/catalogue/its-only-the-himalayas_981/index.html>
{'title': "It's Only the Himalayas", 'price': '£45.17', 'url': 'http://books.toscrape.com/catalogue/its-only-the-himalayas_981/index.html'}
2023-05-02 13:23:20 [scrapy.core.scraper] DEBUG: Scraped from <200 http://books.toscrape.com/catalogue/libertarianism-for-beginners_982/index.html>
{'title': 'Libertarianism for Beginners', 'price': '£51.33', 'url': 'http://books.toscrape.com/catalogue/libertarianism-for-beginners_982/index.html'}
2023-05-02 13:23:20 [scrapy.core.scraper] DEBUG: Scraped from <200 http://books.toscrape.com/catalogue/mesaerion-the-best-science-fiction-stories-1800-1849_983/index.html>
{'title': 'Mesaerion: The Best Science Fiction Stories 1800-1849', 'price': '£37.59', 'url': 'http://books.toscrape.com/catalogue/mesaerion-the-best-science-fiction-stories-1800-1849_983/index.html'}
2023-05-02 13:23:20 [scrapy.core.scraper] DEBUG: Scraped from <200 http://books.toscrape.com/catalogue/olio_984/index.html>
{'title': 'Olio', 'price': '£23.88', 'url': 'http://books.toscrape.com/catalogue/olio_984/index.html'}
2023-05-02 13:23:20 [scrapy.core.scraper] DEBUG: Scraped from <200 http://books.toscrape.com/catalogue/our-band-could-be-your-life-scenes-from-the-american-indie-underground-1981-1991_985/index.html>
{'title': 'Our Band Could Be Your Life: Scenes from the American Indie Underground, 1981-1991', 'price': '£57.25', 'url': 'http://books.toscrape.com/catalogue/our-band-could-be-your-life-scenes-from-the-american-indie-underground-1981-1991_985/index.html'}
2023-05-02 13:23:21 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://books.toscrape.com/catalogue/how-music-works_979/index.html> (referer: http://books.toscrape.com/catalogue/page-2.html)
2023-05-02 13:23:21 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://books.toscrape.com/catalogue/chase-me-paris-nights-2_977/index.html> (referer: http://books.toscrape.com/catalogue/page-2.html)
2023-05-02 13:23:21 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://books.toscrape.com/catalogue/in-her-wake_980/index.html> (referer: http://books.toscrape.com/catalogue/page-2.html)
2023-05-02 13:23:21 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://books.toscrape.com/catalogue/aladdin-and-his-wonderful-lamp_973/index.html> (referer: http://books.toscrape.com/catalogue/page-2.html)

To learn more about link extractors and rule-based crawling, read the following pages of Scrapy documentation:

Intro to Bash scripting for scraping and automation

Tue, 04 Apr 2023 00:00:00 +0000

Bash is a Linux/UNIX program that reads users commands from the users, parses them and executes the appropriate programs through OS-specific APIs. Since it covers these APIs and provides some extra features on top of them this kind of program is called a shell. Bash is not merely an interface between keyboard and exec(2) et. al. It is also a scripting language and interpreter.

Today we are going to explore various scripting facilities in Bash to learn about programmable nature of this shell. The objective here is not to give a tutorial on writing Bash scripts. Instead, I aim to show what scripting capabilities it has and how does it fit into the broader reality, with emphasis on scraping data and automating stuff.

The following text assumes a basic familiarity with the world of Linux/Unix software.

In this world, each program has three communication channels available by default:

Standard input (file descriptor 0)
Standard output (FD 1)
Standard error (FD 2)

When you run a CLI program in a terminal it uses standard input to read stuff you type into it and standard output to print text for you to read. Standard error is typically used for error messages and in some cases for a log output.

Use > to redirect standard output, 2> to redirect standard error and < to redirect standard input, as seen in the following example:

$ ls /etc > etc_list.txt
$ ls /notfound 2> err.msg
$ wc -l < err.msg 
       1

To daisy-chain two or more programs into pipeline such that standard output of one program feeds into standard input of another you can use the pipe operator:

$ fortune | cowsay
 ____________________________ 
/ Sturgeon's Law:            \
|                            |
\ 90% of everything is crud. /
 ---------------------------- 
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

This entails a first baby-step towards using Bash as programming language. But to do programming, we need variables. We can indeed declare them in Bash:

$ x=1
$ echo "$x"
1

To declare a variable, we just did an assign statement. Note that there are no whitespace characters around the = sign - Bash would not be able to correctly parse it otherwise. To see what a value the variable has, we prepended the variable name with a dollar sign and used an in-build echo command.

All variables in Bash are global to the script or user session unless declared with a local keyword within a funcion (e.g. local b=2). Bash does not have any real type system. Pretty much all data is treated as strings with exception of some numeric operations on integers. On it’s own, Bash does not even support floating point operations - you may need to use bc(1) or something else for that.

One way to perform integer arithmetic is to use built-in expr command (make sure that you put whitespaces around parts of expression as otherwise it will be treated as string):

$ expr 5 + 11
16

Another way is to use a let command that will assign the result to a variable:

$ let x=1+3
$ echo "$x"
4
$ let four=2*2
$ echo "$four"
4

let command also supports increments and shorthand (+= / -=):

$ let x++
$ echo "$x"
5
$ let x+=20
$ echo "$x"
25

We can also reference another variable in the computation:

$ let "y=2*x"
$ echo "$y"
10

Yet another way to do integer arithmetic operations is to use the double parenthese notation:

$ two=$((1+1))
$ echo "$two"
2
$ (( three = two + 1 ))
$ echo "$three"
3

Like the let command, this notation also supports increments/decrements and shorthand operations:

$ z=11
$ ((z++))
$ echo "$z"
12
$ ((z+=55))
$ echo "$z"
67
$ ((z--))
$ echo "$z"
66

To compute some floating point numbers, we can use printf command to generate a command for bc(1) and read the result into variable by using a command substitution syntax ($( ... )):

$ a=3
$ b="2.1111111111111111"
$ result=$(printf "scale=5\n%d / %f\n" "$a" "$b" | bc -l)
$ echo "$result"
1.42105

Bash supports basic string manipulation functionality, such as substitutions and slicing.

For a string manipulation syntax, see the following examples:

$ msg="cypherpunks write code"
$ echo "${#msg}" # Get length.
22
$ echo "${msg:0:6}" # Get substring by starting index and length.
cypher
$ echo "${msg:(-4):4}" # Index can also count to count backwards from end of string.
code
$ echo "${msg%code}" # Remove suffix.
cypherpunks write 
$ echo "${msg%cypher}" # Remove prefix.
cypherpunks write code
$ echo "${msg/write/develop}" # Replace string (single match).
cypherpunks develop code
$ echo "${msg//c/C}" # Replace string (multiple matches).
Cypherpunks write Code
$ echo "${msg/%code/software}" # Replace suffix.
cypherpunks write software
$ echo "${msg/#cypher/crypto}" # Replace prefix.
cryptopunks write code

For conditional logic, Bash supports if-statements and case (switch) statements.

In simple cases, if-statements can have one line form:

$ if [[ 1 == 1 ]]; then echo "yes"; fi
yes
$ if [[ 1 == 2 ]]; then echo "yes"; else echo "no"; fi
no

Okay, so this is getting a rather dry and boring, so let’s provide an example from real world software - Axiom.

There are following lines of code in file interact/axiom-wait:

if [ ! -z "$2" ]
then
    instance="$2"
fi

We see a bit different conditional syntax here. $2 is the second CLI argument to the script and here it is being checked for having a defined value. -z stands for “is empty/undefined” and ! is a logical NOT operation.

How does the case statement look like? We can find an example in file interact/axiom-scan:

BASEOS="$(uname)"
case $BASEOS in
'Darwin')
    PATH="$(brew --prefix coreutils)/libexec/gnubin:$PATH"
    ;;
*) ;;
esac

Here it reads the output of uname(1) into variable BASEOS. If the value is Darwin is it updates the PATH environment variable with one more entry. In the default case it does nothing (see line *) ;;).

Like many programming languages, Bash enables iteratative code through for and while loops.

The following example shows how for loop can be used to iterate across output of some command, one line at a time:

i=1
for f in $(find "$tmp/split/" -type f | tr '/' ' ' | awk '{ print $NF }')
do
        instance="$(echo $instances | awk "{ print \$$i }")"
        i=$((i+1))

        mv "$tmp/split/$f" "$tmp/input/$instance"
    done
    total=$i
}

Example of while loop can be seen in file interact/account-helpers/aws.sh:

echo -e -n "${Green}Please enter your AWS Access Key ID (required): \n>> ${Color_Off}"
read ACCESS_KEY
while [[ "$ACCESS_KEY" == "" ]]; do
	echo -e "${BRed}Please provide a AWS Access KEY ID, your entry contained no input.${Color_Off}"
	echo -e -n "${Green}Please enter your token (required): \n>> ${Color_Off}"
	read ACCESS_KEY
done

In this case the loop would iterate until the user would provide a non-empty input (read command reads text from the user and saves it into a variable).

Like many programming languages, Bash provides two basic built-in data structures: arrays and dictionaries.

Bash array can be declared and used as follows:

$ numbers=(1 2 3 4 5) # Declaring an array.
$ echo "${numbers[0]}" # Array indexing - getting an element
1
$ echo "${#numbers[@]}" # Array length
5
$ echo "${numbers[@]}" # Get all elements
1 2 3 4 5
$ numbers[0]=0 # Array indexing - setting an element
$ echo "${numbers[@]}" # Get all elements
0 2 3 4 5
$ numbers+=("six") # Append to array

The following listing shows how Bash dictionaries can be used in practice:

$ declare -A people # Declare empty dictionary.
$ people=(["John"]: "555-1212" ["Steve"]: "555-1313") # Initialise the dictionary
$ echo "${people['John']}" # Get value by key.
555-1212
$ echo "${people['Alice']}"

$ echo "${people[@]}" # Get all keys.
555-1313 555-1212
$ echo "${!people[@]}" # Get all values.
Steve John
$ for key in "${!people[@]}"; do value="${people[$key]}"; echo "$key - $value"; done # Iterate across dictionary.
Steve - 555-1313
John - 555-1212
$ people['Alice']='555-0000' # Set value for key.
$ echo "${people['Alice']}"
555-0000

Note, however, that Bash dictionaries are not supported in legacy (pre-4.0) version of Bash. If you are using macOS then you may need to update your Bash via Homebrew, as Apple does not ship modern Bash versions due to license issue.

Bash also supports functions. One example would be the following simple function from Axiom codebase, file interact/includes/system-notification.sh:

# method issuing notifications in place of notify-send on OSX
# expects a title as first and a notification as second argument
function notifyOSX {
  osascript -e "display notification \"$2\" with title \"$1\""
}

This function generates macOS system notification with title from first argument ($1) and description from second argument ($2). It can be called as follows:

notifyOSX "Warning" "The end is near"

As we can see, the function call expression has the same syntax as a regular Bash command. After, CLI is API when you’re developing shell scripts.

The following CLI tools might be of interest when developing Bash scripts in web scraping/automation/security fields:

xidel for introducing web scraping functionality into your script. Fetches pages and runs XPath queries or CSS selectors for data extraction.
jq - JSON processing tool with it’s own domain-specific language.
pup for running CSS selectors on HTML pages.
Shellcheck is a shell script static analysis tool for finding potential problems with your code. Highly recommended, as Bash has it’s own share of footguns.

We have surveyed the various features of Bash and now can consider the value proposition that Bash brings to the table for someone doing web scraper development, automation engineering, bounty hunting or even old school kind of system administration.

The way I see it, the value proposition can be viewed like this…

First, it provides a common denominator scripting language for Linux/Unix that also serves as a power tool for users working in command line environment. For example, instead of manually running 100 commands to import 100 CSV files into a SQLite database you could do a quick for loop right in your shell. Due to availability of Bash in many server and development environments this will most likely be possible without installing any additional dependencies.

Second, Bash is a glue language for uniting multiple CLI tools into greater workflow. For example, you may have a Scrapy project that you run through Scrapy CLI and also some helper scripts to postprocess your data before uploading it to some remote system. Due to Bash statements being pretty much the same as Unix/Linux commands (with some extra features) you have a low friction way to chain the invocations of all your tools into single flow that is launched from a Bash script.

Axiom project demonstrates that one can develop quite extensive piece of software by mainly relying on Bash. However, I would not recommend doing so as there are better languages for this kind of work. The value of Bash scripting lies in introducing programmability and automation capabilities into command line workflows.

To learn more about Bash scripting, see the following resources:

Don't JSFuck with me: Part 2

Tue, 14 Mar 2023 00:00:00 +0000

In the previous post we went through using Babel AST transforms to simplify JSFuck-generated unary and binary expressions. This was shown to undo a lot, but not all of the obfuscation. What we still have to do is to deal with certain API and runtime hacks that JSFuck leverages to obfuscate some of the characters that are not covered just by abusing type coercion and atomic components of JS (string and array indexing, logical/arithmetic operations).

Suppose we want to encode a { character with JSFuck. Putting this into JSFuck web app (make sure that checkboxes for additional features are not checked) yields the following code:

(!![]+[][(![]+[])[+[]]+(![]+[])[!+[]+!+[]]+(![]+[])[+!+[]]+(!![]+[])[+[]]])[!+[]+!+[]+[+[]]]

Screenshot 1

If we put it into ASTExplorer we get pretty much the same situation as before - a lot of BinaryExpression, UnaryExpression, ArrayExpression and MemberExpression nodes making up the tree.

Screenshot 2

We can try using our previously developed transform to simplify the code.

Applying it three times gives us the following results:

(true + []["false"[0] + "false"[2] + "false"[1] + "true"[0]])["20"];
(true + []["f" + "l" + "a" + "t"])[20];
(true + []["flat"])[20];

The transform we have from earlier cannot simplify the code any further. What we got here matches line 108 in jsfuck.js.

The above trick works by coercing Array.prototype.flat() to string by adding true to it (addition operation prefers to convert stuff into strings) and indexing the resulting string:

$ node
Welcome to Node.js v19.0.1.
Type ".help" for more information.
> []["flat"]
[Function: flat]
> String([]["flat"])
'function flat() { [native code] }'
> true + []["flat"]
'truefunction flat() { [native code] }'
> (true + []["flat"])[20]
'{'

So what we can do at AST level is pattern-match on []["flat"] part, convert it to string equivalent and replace it with equivalent StringLiteral node in the AST so that expression evaluation code can simplify it further. We also should check if []["flat"] is part of binary expression. This can be done with a following transform:

export default function (babel) {
  const { types: t } = babel;

  return {
    name: "undo-flat-trick", // not required
    visitor: {
      MemberExpression(path) {
        let node = path.node;
        if (!t.isArrayExpression(node.object)) return;
        if (node.object.elements.length != 0) return;
        if (!t.isStringLiteral(node.property)) return;
        if (node.property.value === "flat" && t.isBinaryExpression(path.parent)) {
          path.replaceWith(t.stringLiteral(String(Array.prototype.flat)));
        }
      }
    }
  };
}

It converts the snippet we had to the following:

(true + "function flat() { [native code] }")[20];

Our previous transform can pick it up from here, as we only have addition and string indexing.

Similarly to the flat() trick, JSFuck abuses entries() method, as can be seen in the following MAPPING object key-value pairs:

    'b':   '([]["entries"]()+"")[2]',
    'j':   '([]["entries"]()+"")[3]',
    'A':   '(NaN+[]["entries"]())[11]',
    '[':   '([]["entries"]()+"")[0]',
    ']':   '([]["entries"]()+"")[22]',

The way to undo this is equivalent to dealing with flat() trick:

export default function (babel) {
  const { types: t } = babel;

  return {
    name: "undo-entries-trick", // not required
    visitor: {
      CallExpression(path) {
        let node = path.node;
        if (!t.isMemberExpression(node.callee)) return;
        if (!t.isArrayExpression(node.callee.object)) return;
        if (node.callee.object.elements.length != 0) return;
        if (!t.isStringLiteral(node.callee.property)) return;
        if (node.callee.property.value === "entries" && t.isBinaryExpression(path.parent)) {
          path.replaceWith(t.stringLiteral(String(Array.prototype.entries)));
        }
      }
    }
  };
}

Another thing that JSFuck does is abusing array’s concat() method to obfuscate a comma character:

    ',':   '[[]]["concat"]([[]])+""',

This abuses how array string representation is computed, as evident by a quick experiment in Node.JS REPL:

> [[]]["concat"]([[]])
[ [], [] ]
> String([[]]["concat"]([[]]))
','
> String([ [], [] ])
','
> [[]]["concat"]([[]]) + ""
','

Once again, we pattern-match the part that is to be converted to string due to type coercion, create a StringLiteral out of it and replace it into the AST:

export default function (babel) {
  const { types: t } = babel;
  
  return {
    name: "undo-concat-trick", // not required
    visitor: {
      CallExpression(path) {
        let node = path.node;
        
        if (!t.isMemberExpression(node.callee)) return;
        if (!t.isArrayExpression(node.callee.object)) return;
        if (node.callee.object.elements.length != 1) return;
        if (!t.isArrayExpression(node.callee.object.elements[0])) return;
        if (node.callee.object.elements[0].elements.length != 0) return;
        
        if (!t.isStringLiteral(node.callee.property)) return;
        if (node.callee.property.value != "concat") return;
        
        if (!t.isArrayExpression(node.arguments[0])) return;
        if (node.arguments[0].elements.length != 1) return;
        if (!t.isArrayExpression(node.arguments[0].elements[0])) return;
        if (node.arguments[0].elements[0].elements.length != 0) return;
        
        if (!t.isBinaryExpression(path.parent)) return;
        
        path.replaceWith(t.valueToNode(String([ [], [] ])));
      }
    }
  };
}

In AST Explorer we can verify that [[]]["concat"]([[]])+"" is converted to "," + "";.

String.prototype.slice() method is also leveraged by JSFuck:

    'D':   'Function("return escape")()([]["flat"])["slice"]("-1")',
    '}':   '([]["flat"]+"")["slice"]("-1")',

We see that these cases also rely on some further trickery. Since we already know how to deal with flat() trick, we will focus on the latter on for now.

Running the transform to undo flat() trick converts it to the following:

("function flat() { [native code] }" + "")["slice"]("-1");

According to the documentation, calling slice() on a string with single negative number returns a trailing substring. In this case it will be single-character substring, i.e. just the last character.

We develop another AST transform to address the slice() trick:

export default function (babel) {
  const { types: t } = babel;

  return {
    name: "undo-slice-trick", // not required
    visitor: {
      CallExpression(path) {
        let node = path.node;
        if (!t.isMemberExpression(node.callee)) return;
        if (!t.isStringLiteral(node.callee.object)) return;
        if (!t.isStringLiteral(node.callee.property)) return;
        if (node.arguments.length != 1) return;
        if (node.callee.property.value === "slice" && 
            (t.isStringLiteral(node.arguments[0]) || 
             t.isNumericLiteral(node.arguments[0])) &&
             Number(node.arguments[0].value) === -1) {
          let fromStr = node.callee.object.value;
          path.replaceWith(t.valueToNode(fromStr.slice(-1)));
        }
      }
    }
  };
}

Nothing particulary different here, except that the subtree matching part had to check for some other things when matching.

But before running this transform we have to deal with string concatenation, as the matching is based on StringLiteral node being at callee property of a CallExpression. Applying the expression evaluation code simplifies it to the form that is required by the new transform we just wrote:

"function flat() { [native code] }"["slice"]("-1");

Now, when we apply the new transform we get the original character.

Letter B is obfuscated by leveraging Boolean type name:

    'B':   '(+[]+Boolean)[10]',

Running the expression evaluation code on the obfuscated snippet converts it to:

(0 + Boolean)[10];

Now we can undo the Boolean trick, which is quite trivial:

export default function (babel) {
  const { types: t } = babel;

  return {
    name: "undo-boolean-trick", // not required
    visitor: {
      Identifier(path) {
        let node = path.node;
       	if (!t.isBinaryExpression(path.parent)) return;
        if (path.parent.operator != "+") return;
        if (node.name === "Boolean") path.replaceWith(t.valueToNode(String(Boolean)));
      }
    }
  };
}

Running this one gives us the following snippet:

(0 + "function Boolean() { [native code] }")[10];

Running expression evaluation transform twice simplifies this to B character:

"0function Boolean() { [native code] }"[10];
"B";

JSFuck also abuses a fontcolor() function to encode some characters:

    'q':   '("")["fontcolor"]([0]+false+")[20]',
    '"':   '("")["fontcolor"]()[12]',
    '&':   '("")["fontcolor"](")[13]',
    ';':   '("")["fontcolor"](NaN+")[21]',
    '=':   '("")["fontcolor"]()[11]',

This trick actually relies on the output of fontcolor() function:

> ("")["fontcolor"]()[11]
'='
> ("")["fontcolor"]()
'<font color="undefined"></font>'
> String.prototype.fontcolor("")
'<font color=""></font>'
> String.prototype.fontcolor()[11]
'='

AST transform to undo this would be as follows:

export default function (babel) {
  const { types: t } = babel;

  return {
    name: "undo-fontcolor-trick", // not required
    visitor: {
      CallExpression(path) {
        let node = path.node;
        if (!t.isMemberExpression(node.callee)) return;
        if (!t.isStringLiteral(node.callee.object)) return;
        if (!t.isStringLiteral(node.callee.property)) return;
        if (node.callee.property.value === "fontcolor") {
           if (node.arguments.length === 0) {
             path.replaceWith(t.valueToNode(String.prototype.fontcolor()));
           } else if (t.isLiteral(node.arguments[0])) {
             path.replaceWith(t.valueToNode(String.prototype.fontcolor(node.arguments[0].value)))
           }
        }
      }
    }
  };
}

("")["fontcolor"]()[12] (snippet for double-quote character) is simplified to ""[12];.

Let’s go through a somewhat bigger example.

Obfuscating a semicolon character yields the following code:

([]+[])[(![]+[])[+[]]+(!![]+[][(![]+[])[+[]]+(![]+[])[!+[]+!+[]]+(![]+[])[+!+[]]+(!![]+[])[+[]]])[+!+[]+[+[]]]+([][[]]+[])[+!+[]]+(!![]+[])[+[]]+([][(![]+[])[+[]]+(![]+[])[!+[]+!+[]]+(![]+[])[+!+[]]+(!![]+[])[+[]]]+[])[!+[]+!+[]+!+[]]+(!![]+[][(![]+[])[+[]]+(![]+[])[!+[]+!+[]]+(![]+[])[+!+[]]+(!![]+[])[+[]]])[+!+[]+[+[]]]+(![]+[])[!+[]+!+[]]+(!![]+[][(![]+[])[+[]]+(![]+[])[!+[]+!+[]]+(![]+[])[+!+[]]+(!![]+[])[+[]]])[+!+[]+[+[]]]+(!![]+[])[+!+[]]](+[![]]+([]+[])[(![]+[])[+[]]+(!![]+[][(![]+[])[+[]]+(![]+[])[!+[]+!+[]]+(![]+[])[+!+[]]+(!![]+[])[+[]]])[+!+[]+[+[]]]+([][[]]+[])[+!+[]]+(!![]+[])[+[]]+([][(![]+[])[+[]]+(![]+[])[!+[]+!+[]]+(![]+[])[+!+[]]+(!![]+[])[+[]]]+[])[!+[]+!+[]+!+[]]+(!![]+[][(![]+[])[+[]]+(![]+[])[!+[]+!+[]]+(![]+[])[+!+[]]+(!![]+[])[+[]]])[+!+[]+[+[]]]+(![]+[])[!+[]+!+[]]+(!![]+[][(![]+[])[+[]]+(![]+[])[!+[]+!+[]]+(![]+[])[+!+[]]+(!![]+[])[+[]]])[+!+[]+[+[]]]+(!![]+[])[+!+[]]]()[+!+[]+[!+[]+!+[]]])[!+[]+!+[]+[+!+[]]]

We can apply the expression evaluation transform until we hit the flat() trick:

""["false"[0] + (true + []["false"[0] + "false"[2] + "false"[1] + "true"[0]])["10"] + (undefined + [])[1] + "true"[0] + ([]["false"[0] + "false"[2] + "false"[1] + "true"[0]] + [])[3] + (true + []["false"[0] + "false"[2] + "false"[1] + "true"[0]])["10"] + "false"[2] + (true + []["false"[0] + "false"[2] + "false"[1] + "true"[0]])["10"] + "true"[1]](0 / 0 + ""["false"[0] + (true + []["false"[0] + "false"[2] + "false"[1] + "true"[0]])["10"] + (undefined + [])[1] + "true"[0] + ([]["false"[0] + "false"[2] + "false"[1] + "true"[0]] + [])[3] + (true + []["false"[0] + "false"[2] + "false"[1] + "true"[0]])["10"] + "false"[2] + (true + []["false"[0] + "false"[2] + "false"[1] + "true"[0]])["10"] + "true"[1]]()["12"])["21"];
""["f" + (true + []["f" + "l" + "a" + "t"])[10] + "undefined"[1] + "t" + ([]["f" + "l" + "a" + "t"] + [])[3] + (true + []["f" + "l" + "a" + "t"])[10] + "l" + (true + []["f" + "l" + "a" + "t"])[10] + "r"](NaN + ""["f" + (true + []["f" + "l" + "a" + "t"])[10] + "undefined"[1] + "t" + ([]["f" + "l" + "a" + "t"] + [])[3] + (true + []["f" + "l" + "a" + "t"])[10] + "l" + (true + []["f" + "l" + "a" + "t"])[10] + "r"]()[12])[21];
""["f" + (true + []["flat"])[10] + "n" + "t" + ([]["flat"] + [])[3] + (true + []["flat"])[10] + "l" + (true + []["flat"])[10] + "r"](NaN + ""["f" + (true + []["flat"])[10] + "n" + "t" + ([]["flat"] + [])[3] + (true + []["flat"])[10] + "l" + (true + []["flat"])[10] + "r"]()[12])[21];

Applying the previous transform to undo this provides us with path to further apply expression evaluation:

""["f" + (true + "function flat() { [native code] }")[10] + "n" + "t" + ("function flat() { [native code] }" + [])[3] + (true + "function flat() { [native code] }")[10] + "l" + (true + "function flat() { [native code] }")[10] + "r"](NaN + ""["f" + (true + "function flat() { [native code] }")[10] + "n" + "t" + ("function flat() { [native code] }" + [])[3] + (true + "function flat() { [native code] }")[10] + "l" + (true + "function flat() { [native code] }")[10] + "r"]()[12])[21];
""["f" + "truefunction flat() { [native code] }"[10] + "n" + "t" + "function flat() { [native code] }"[3] + "truefunction flat() { [native code] }"[10] + "l" + "truefunction flat() { [native code] }"[10] + "r"](NaN + ""["f" + "truefunction flat() { [native code] }"[10] + "n" + "t" + "function flat() { [native code] }"[3] + "truefunction flat() { [native code] }"[10] + "l" + "truefunction flat() { [native code] }"[10] + "r"]()[12])[21];
""["f" + "o" + "n" + "t" + "c" + "o" + "l" + "o" + "r"](NaN + ""["f" + "o" + "n" + "t" + "c" + "o" + "l" + "o" + "r"]()[12])[21];
""["fontcolor"](NaN + ""["fontcolor"]()[12])[21];

Now it’s time to apply the undo-fontcolor-trick transform:

""["fontcolor"](NaN + ""[12])[21];

We do two more iterations of expression evaluation:

""["fontcolor"](NaN + "\"")[21];
""["fontcolor"]("NaN\"")[21];

… and apply undo-fontcolor-trick transform once again:

""[21];

Applying expression evaluation one more time gives us the original character:

";";

JSFuck abuses italics() in a similar way that is also based on indexing the output string:

    '/':   '(false+[0])["italics"]()[10]',
    '<':   '("")["italics"]()[0]',
    '>':   '("")["italics"]()[2]',
    'C':   'Function("return escape")()(("")["italics"]())[2]',

The transform to undo the trick is similar to the one we did for fontcolor() case:

export default function (babel) {
  const { types: t } = babel;

  return {
    name: "undo-italics-trick", // not required
    visitor: {
      CallExpression(path) {
        let node = path.node;
        if (!t.isMemberExpression(node.callee)) return;
        if (!t.isStringLiteral(node.callee.object)) return;
        if (!t.isStringLiteral(node.callee.property)) return;
        if (node.callee.property.value === "italics") {
           const fromStr = node.callee.object.value;
           if (node.arguments.length === 0) {
             path.replaceWith(t.valueToNode(fromStr.italics()));
           } 
        }
      }
    }
  };
}

Quick check on AST Explorer shows that ("")["italics"]()[0] is simplified to ""[0];, which is something we can process further with the other code.

When encoding character C, JSFuck also does another trick that relies on using escape() function to URL-encode some string, then indexing the output:

> Function("return escape")()(("")["italics"]())[2]
'C'
> Function("return escape")()(("")["italics"]())
'%3Ci%3E%3C/i%3E'
> ("")["italics"]()
'<i></i>'
> Function("return escape")()('<i></i>')
'%3Ci%3E%3C/i%3E'
> Function("return escape")()('<i></i>')[2]
'C'
> escape('<i></i>')
'%3Ci%3E%3C/i%3E'
> escape('<i></i>')[2]
'C'

Besides C character this trick is also used to obfuscated D and % characters (in combination with some other tricks that we covered already):

    'C':   'Function("return escape")()(("")["italics"]())[2]',
    'D':   'Function("return escape")()([]["flat"])["slice"]("-1")',
    '%':   'Function("return escape")()([]["flat"])[21]',

Applying the undo-italics-trick to the snippet from the first case turns it into:

Function("return escape")()("<i></i>")[2];

Now we need an AST transform that calls escape() function and substitutes a URL-encoded string into the AST. We also need to deal with a special case of the flat() trick. The previous transform will not work as-is due to being developed for cases with binary expressions.

So the AST transform to deal with this stuff would be like this:

export default function (babel) {
  const { types: t } = babel;

  return {
    name: "undo-function-escape-trick", // not required
    visitor: {
      MemberExpression(path) {
        let node = path.node;
        if (!t.isArrayExpression(node.object)) return;
        if (node.object.elements.length != 0) return;
        if (!t.isStringLiteral(node.property)) return;
        if (node.property.value === "flat") path.replaceWith(t.valueToNode(String(Array.prototype.flat)));
      },
      CallExpression(path) {
        let node = path.node;
        if (t.isCallExpression(node.callee) &&
            t.isCallExpression(node.callee.callee) &&
            t.isIdentifier(node.callee.callee.callee) &&
            node.callee.callee.callee.name === "Function" &&
            node.callee.callee.arguments.length === 1 &&
            t.isStringLiteral(node.callee.callee.arguments[0]) &&
            node.callee.callee.arguments[0].value === "return escape" &&
            t.isStringLiteral(node.arguments[0])
           ) {
            path.replaceWith(t.valueToNode(escape(node.arguments[0].value)));
        }
      }
    }
  };
}

Let us try it on the snippet for % character - Function("return escape")()([]["flat"])[21]. We need to do it twice.

Function("return escape")()("function flat() { [native code] }")[21];
"function%20flat%28%29%20%7B%20%5Bnative%20code%5D%20%7D"[21];

Expression evaluation transform pulls out the % character from this.

For 3 of the characters, JSFuck relies on calling Date() to obfuscate them:

    'G':   '(false+Function("return Date")()())[30]',
    'M':   '(true+Function("return Date")()())[30]',
    'T':   '(NaN+Function("return Date")()())[30]',

This relies on characters G, M and T being always present in the output of Date() function at predictable indices. Some dummy values are added to tweak the output string a little, then indexing it to pull out the needed character.

The transform code to undo this is similar to the ones we wrote before:

export default function (babel) {
  const { types: t } = babel;

  return {
    name: "undo-function-date-trick", // not required
    visitor: {
      CallExpression(path) {
        let node = path.node;
        if (t.isCallExpression(node.callee) &&
            t.isCallExpression(node.callee.callee) &&
            t.isIdentifier(node.callee.callee.callee) &&
            node.callee.callee.callee.name === "Function" &&
            node.callee.callee.arguments.length === 1 &&
            t.isStringLiteral(node.callee.callee.arguments[0]) &&
            node.callee.callee.arguments[0].value === "return Date"
           ) {
            path.replaceWith(t.valueToNode(Date()));
        }
      }
    }
  };
}

This converts Function("return Date")()() (equivalent to Date()) part to string literal with datetime string, which can be further dealt with by the expression evaluation transform.

JSFuck can also leverage RegExp API for obfuscation, as evident in the following cases:

    'E':   '(RegExp+"")[12]',
    'R':   '(+[]+RegExp)[10]',
    ':':   '(RegExp()+"")[3]',
    '?':   '(RegExp()+"")[2]',
    '\\':  '(RegExp("/")+"")[1]',

As you can see, in some cases RegExp identifier is used in addition operation but in other cases, a return value from RegExp() is used instead. We should cover both forms when deobfuscating.

These can be addressed with the following transform:

export default function (babel) {
  const { types: t } = babel;

  return {
    name: "undo-regexp-trick", // not required
    visitor: {
      Identifier(path) {
        let node = path.node;
       	if (!t.isBinaryExpression(path.parent)) return;
        if (path.parent.operator != "+") return;
        if (node.name === "RegExp") path.replaceWith(t.valueToNode(String(RegExp)));
      },
      CallExpression(path) {
        let node = path.node;
        if (!t.isBinaryExpression(path.parent)) return;
        if (path.parent.operator != "+") return;
        if (!t.isIdentifier(node.callee)) return;
        if (node.callee.name != "RegExp") return;
        if (node.arguments.length === 0) {
          path.replaceWith(t.valueToNode(String(RegExp())));
        } else if (node.arguments.length === 1 &&
                   t.isStringLiteral(node.arguments[0]) &&
                   node.arguments[0].value === "/") {
          path.replaceWith(t.valueToNode(String(RegExp("/"))));
        }
      }
    }
  };
}

There are few more string-based tricks that JSFuck does. One is by leveraging the type name of String in the following cases:

    'g':   '(false+[0]+String)[20]',
    'S':   '(+[]+String)[10]',

Getting these cases to the point of tractability by expression evaluation transform is easy and can be done with a transform very similar to the one for Boolean case earlier in this post:

export default function (babel) {
  const { types: t } = babel;

  return {
    name: "undo-string-trick", // not required
    visitor: {
      Identifier(path) {
        let node = path.node;
       	if (!t.isBinaryExpression(path.parent)) return;
        if (path.parent.operator != "+") return;
        if (node.name === "String") path.replaceWith(t.valueToNode(String(String)));
      }
    }
  };
}

Another trick we need to address is abusing toString() method as seen in the following lines from jsfuck.js:

    'h':   '(+(101))["to"+String["name"]](21)[1]',
    'k':   '(+(20))["to"+String["name"]](21)',
    'p':   '(+(211))["to"+String["name"]](31)[1]',
    'v':   '(+(31))["to"+String["name"]](32)',
    'w':   '(+(32))["to"+String["name"]](33)',
    'x':   '(+(101))["to"+String["name"]](34)[1]',
    'z':   '(+(35))["to"+String["name"]](36)',
    'U':   '(NaN+Object()["to"+String["name"]]["call"]())[11]',

With the exception of last case, all of these rely on calling toString() method on a number to covert it to numeric string with non-standard base. In some cases this numeric string is indexed to pull out the needed character.

Let us unpack the case for h in Node.JS REPL:

> (+(101))["to"+String["name"]](21)[1]
'h'
> String["name"]
'String'
> (+(101))
101
> "to"+String["name"]
'toString'
> 101["toString"](21)
'4h'
> 101["toString"](21)[1]
'h'

Applying the expression evaluation transform to the h case only simplifies the unary expression:

101["to" + String["name"]](21)[1];

Before we proceed with number-to-string conversion here, let us write a quick helper transform that simplifies String["name"] part:

export default function (babel) {
  const { types: t } = babel;

  return {
    name: "simplify-string-name", // not required
    visitor: {
      MemberExpression(path) {
        let node = path.node;
        if (!t.isIdentifier(node.object)) return;
        if (!t.isStringLiteral(node.property)) return;
        if (node.object.name === "String" && node.property.value === "name") {
          path.replaceWith(t.valueToNode("String"));
        }
      }
    }
  };
}

This simplifies the snippet to:

101["to" + "String"](21)[1];

Applying expression evaluation transform on this simplifies it bit further:

101["toString"](21)[1];

Once again, we pattern match on the relevant subtree and apply the change on it.

See the following transform:

export default function (babel) {
  const { types: t } = babel;

  return {
    name: "undo-number-tostring-trick", // not required
    visitor: {
      CallExpression(path) {
        let node = path.node;
        if (!t.isMemberExpression(node.callee)) return;
        if (!t.isNumericLiteral(node.callee.object)) return;
        if (!t.isStringLiteral(node.callee.property)) return;
        if (node.callee.property.value != "toString") return;
        if (node.arguments.length != 1) return;
        if (!t.isLiteral(node.arguments[0])) return;
        
        let numericStr = node.callee.object.value["toString"](node.arguments[0].value);
        path.replaceWith(t.valueToNode(numericStr));
      }
    }
  };
}

This turns our current snippet to:

"4h"[1];

Now let us deal with the snippet for character U:

(NaN+Object()["to"+String["name"]]["call"]())[11]

Applying our helper transform simplifies it to:

(NaN + Object()["to" + "String"]["call"]())[11];

We can further simplify it by applying expression evaluation transform:

(NaN + Object()["toString"]["call"]())[11];

Now we need another transform written that matches an exact subtree and simplifies it further. It would look like this:

export default function (babel) {
  const { types: t } = babel;
  
  return {
    name: "undo-object-tostring-call", // not required
    visitor: {
      CallExpression(path) {
        let node = path.node;
        let callee = node.callee;
        if (!t.isMemberExpression(callee)) return;
        let calleeObj = callee.object;
        if (!t.isMemberExpression(calleeObj)) return;
        if (!t.isCallExpression(calleeObj.object)) return;
        calleeObj = calleeObj.object;
        if (!t.isIdentifier(calleeObj.callee)) return;
        if (calleeObj.callee.name != "Object") return;
        if (!t.isStringLiteral(callee.property)) return;
        if (callee.property.value != "call") return;
        if (!t.isStringLiteral(callee.object.property)) return;
        if (callee.object.property.value != "toString") return;
        path.replaceWith(t.valueToNode(Object()["toString"]["call"]()));
      }
    }
  };
}

By applying this transform, we get the following snippet:

(NaN + "[object Undefined]")[11];

This can be further simplified to character U by applying expression evaluation transform two more times:

"NaN[object Undefined]"[11];
"U";

We’re getting close to the end, but there’s still some uncovered ground left. In the MAPPING object we see that some characters don’t have matching snippet assignmed to them:

    'H':   null,
    'J':   null,
    'K':   null,
    'L':   null,
    'P':   null,
    'Q':   null,
    'V':   null,
    'W':   null,
    'X':   null,
    'Y':   null,
    'Z':   null,
    '!':   null,
    '#':   null,
    '$':   null,
    '\'':  null,
    '*':   null,
    '@':   null,
    '^':   null,
    '_':   null,
    '`':   null,
    '|':   null,
    '~':   null

Let’s try to find out what happens when one of these characters are being obfuscated. Can we deobfuscate them back?

Obfuscating @ with JSFuck gives us the following code:

[][(![]+[])[+[]]+(![]+[])[!+[]+!+[]]+(![]+[])[+!+[]]+(!![]+[])[+[]]][([][(![]+[])[+[]]+(![]+[])[!+[]+!+[]]+(![]+[])[+!+[]]+(!![]+[])[+[]]]+[])[!+[]+!+[]+!+[]]+(!![]+[][(![]+[])[+[]]+(![]+[])[!+[]+!+[]]+(![]+[])[+!+[]]+(!![]+[])[+[]]])[+!+[]+[+[]]]+([][[]]+[])[+!+[]]+(![]+[])[!+[]+!+[]+!+[]]+(!![]+[])[+[]]+(!![]+[])[+!+[]]+([][[]]+[])[+[]]+([][(![]+[])[+[]]+(![]+[])[!+[]+!+[]]+(![]+[])[+!+[]]+(!![]+[])[+[]]]+[])[!+[]+!+[]+!+[]]+(!![]+[])[+[]]+(!![]+[][(![]+[])[+[]]+(![]+[])[!+[]+!+[]]+(![]+[])[+!+[]]+(!![]+[])[+[]]])[+!+[]+[+[]]]+(!![]+[])[+!+[]]]((!![]+[])[+!+[]]+(!![]+[])[!+[]+!+[]+!+[]]+(!![]+[])[+[]]+([][[]]+[])[+[]]+(!![]+[])[+!+[]]+([][[]]+[])[+!+[]]+([]+[])[(![]+[])[+[]]+(!![]+[][(![]+[])[+[]]+(![]+[])[!+[]+!+[]]+(![]+[])[+!+[]]+(!![]+[])[+[]]])[+!+[]+[+[]]]+([][[]]+[])[+!+[]]+(!![]+[])[+[]]+([][(![]+[])[+[]]+(![]+[])[!+[]+!+[]]+(![]+[])[+!+[]]+(!![]+[])[+[]]]+[])[!+[]+!+[]+!+[]]+(!![]+[][(![]+[])[+[]]+(![]+[])[!+[]+!+[]]+(![]+[])[+!+[]]+(!![]+[])[+[]]])[+!+[]+[+[]]]+(![]+[])[!+[]+!+[]]+(!![]+[][(![]+[])[+[]]+(![]+[])[!+[]+!+[]]+(![]+[])[+!+[]]+(!![]+[])[+[]]])[+!+[]+[+[]]]+(!![]+[])[+!+[]]]()[+!+[]+[!+[]+!+[]]]+([][(![]+[])[+[]]+(![]+[])[!+[]+!+[]]+(![]+[])[+!+[]]+(!![]+[])[+[]]][([][(![]+[])[+[]]+(![]+[])[!+[]+!+[]]+(![]+[])[+!+[]]+(!![]+[])[+[]]]+[])[!+[]+!+[]+!+[]]+(!![]+[][(![]+[])[+[]]+(![]+[])[!+[]+!+[]]+(![]+[])[+!+[]]+(!![]+[])[+[]]])[+!+[]+[+[]]]+([][[]]+[])[+!+[]]+(![]+[])[!+[]+!+[]+!+[]]+(!![]+[])[+[]]+(!![]+[])[+!+[]]+([][[]]+[])[+[]]+([][(![]+[])[+[]]+(![]+[])[!+[]+!+[]]+(![]+[])[+!+[]]+(!![]+[])[+[]]]+[])[!+[]+!+[]+!+[]]+(!![]+[])[+[]]+(!![]+[][(![]+[])[+[]]+(![]+[])[!+[]+!+[]]+(![]+[])[+!+[]]+(!![]+[])[+[]]])[+!+[]+[+[]]]+(!![]+[])[+!+[]]]((!![]+[])[+!+[]]+(!![]+[])[!+[]+!+[]+!+[]]+(!![]+[])[+[]]+([][[]]+[])[+[]]+(!![]+[])[+!+[]]+([][[]]+[])[+!+[]]+(![]+[+[]])[([![]]+[][[]])[+!+[]+[+[]]]+(!![]+[])[+[]]+(![]+[])[+!+[]]+(![]+[])[!+[]+!+[]]+([![]]+[][[]])[+!+[]+[+[]]]+([][(![]+[])[+[]]+(![]+[])[!+[]+!+[]]+(![]+[])[+!+[]]+(!![]+[])[+[]]]+[])[!+[]+!+[]+!+[]]+(![]+[])[!+[]+!+[]+!+[]]]()[+!+[]+[+[]]]+![]+(![]+[+[]])[([![]]+[][[]])[+!+[]+[+[]]]+(!![]+[])[+[]]+(![]+[])[+!+[]]+(![]+[])[!+[]+!+[]]+([![]]+[][[]])[+!+[]+[+[]]]+([][(![]+[])[+[]]+(![]+[])[!+[]+!+[]]+(![]+[])[+!+[]]+(!![]+[])[+[]]]+[])[!+[]+!+[]+!+[]]+(![]+[])[!+[]+!+[]+!+[]]]()[+!+[]+[+[]]])()[([][(![]+[])[+[]]+(![]+[])[!+[]+!+[]]+(![]+[])[+!+[]]+(!![]+[])[+[]]]+[])[!+[]+!+[]+!+[]]+(!![]+[][(![]+[])[+[]]+(![]+[])[!+[]+!+[]]+(![]+[])[+!+[]]+(!![]+[])[+[]]])[+!+[]+[+[]]]+([][[]]+[])[+!+[]]+(![]+[])[!+[]+!+[]+!+[]]+(!![]+[])[+[]]+(!![]+[])[+!+[]]+([][[]]+[])[+[]]+([][(![]+[])[+[]]+(![]+[])[!+[]+!+[]]+(![]+[])[+!+[]]+(!![]+[])[+[]]]+[])[!+[]+!+[]+!+[]]+(!![]+[])[+[]]+(!![]+[][(![]+[])[+[]]+(![]+[])[!+[]+!+[]]+(![]+[])[+!+[]]+(!![]+[])[+[]]])[+!+[]+[+[]]]+(!![]+[])[+!+[]]]((![]+[+[]])[([![]]+[][[]])[+!+[]+[+[]]]+(!![]+[])[+[]]+(![]+[])[+!+[]]+(![]+[])[!+[]+!+[]]+([![]]+[][[]])[+!+[]+[+[]]]+([][(![]+[])[+[]]+(![]+[])[!+[]+!+[]]+(![]+[])[+!+[]]+(!![]+[])[+[]]]+[])[!+[]+!+[]+!+[]]+(![]+[])[!+[]+!+[]+!+[]]]()[+!+[]+[+[]]])+[])[+!+[]]+[+!+[]]+[+[]]+[+[]]+([]+[])[(![]+[])[+[]]+(!![]+[][(![]+[])[+[]]+(![]+[])[!+[]+!+[]]+(![]+[])[+!+[]]+(!![]+[])[+[]]])[+!+[]+[+[]]]+([][[]]+[])[+!+[]]+(!![]+[])[+[]]+([][(![]+[])[+[]]+(![]+[])[!+[]+!+[]]+(![]+[])[+!+[]]+(!![]+[])[+[]]]+[])[!+[]+!+[]+!+[]]+(!![]+[][(![]+[])[+[]]+(![]+[])[!+[]+!+[]]+(![]+[])[+!+[]]+(!![]+[])[+[]]])[+!+[]+[+[]]]+(![]+[])[!+[]+!+[]]+(!![]+[][(![]+[])[+[]]+(![]+[])[!+[]+!+[]]+(![]+[])[+!+[]]+(!![]+[])[+[]]])[+!+[]+[+[]]]+(!![]+[])[+!+[]]]()[+!+[]+[!+[]+!+[]]])()

We put it into ASTExplorer and apply expression evaluation transform several times:

[]["false"[0] + "false"[2] + "false"[1] + "true"[0]][([]["false"[0] + "false"[2] + "false"[1] + "true"[0]] + [])[3] + (true + []["false"[0] + "false"[2] + "false"[1] + "true"[0]])["10"] + (undefined + [])[1] + "false"[3] + "true"[0] + "true"[1] + (undefined + [])[0] + ([]["false"[0] + "false"[2] + "false"[1] + "true"[0]] + [])[3] + "true"[0] + (true + []["false"[0] + "false"[2] + "false"[1] + "true"[0]])["10"] + "true"[1]]("true"[1] + "true"[3] + "true"[0] + (undefined + [])[0] + "true"[1] + (undefined + [])[1] + ""["false"[0] + (true + []["false"[0] + "false"[2] + "false"[1] + "true"[0]])["10"] + (undefined + [])[1] + "true"[0] + ([]["false"[0] + "false"[2] + "false"[1] + "true"[0]] + [])[3] + (true + []["false"[0] + "false"[2] + "false"[1] + "true"[0]])["10"] + "false"[2] + (true + []["false"[0] + "false"[2] + "false"[1] + "true"[0]])["10"] + "true"[1]]()["12"] + ([]["false"[0] + "false"[2] + "false"[1] + "true"[0]][([]["false"[0] + "false"[2] + "false"[1] + "true"[0]] + [])[3] + (true + []["false"[0] + "false"[2] + "false"[1] + "true"[0]])["10"] + (undefined + [])[1] + "false"[3] + "true"[0] + "true"[1] + (undefined + [])[0] + ([]["false"[0] + "false"[2] + "false"[1] + "true"[0]] + [])[3] + "true"[0] + (true + []["false"[0] + "false"[2] + "false"[1] + "true"[0]])["10"] + "true"[1]]("true"[1] + "true"[3] + "true"[0] + (undefined + [])[0] + "true"[1] + (undefined + [])[1] + "false0"[([false] + undefined)["10"] + "true"[0] + "false"[1] + "false"[2] + ([false] + undefined)["10"] + ([]["false"[0] + "false"[2] + "false"[1] + "true"[0]] + [])[3] + "false"[3]]()["10"] + false + "false0"[([false] + undefined)["10"] + "true"[0] + "false"[1] + "false"[2] + ([false] + undefined)["10"] + ([]["false"[0] + "false"[2] + "false"[1] + "true"[0]] + [])[3] + "false"[3]]()["10"])()[([]["false"[0] + "false"[2] + "false"[1] + "true"[0]] + [])[3] + (true + []["false"[0] + "false"[2] + "false"[1] + "true"[0]])["10"] + (undefined + [])[1] + "false"[3] + "true"[0] + "true"[1] + (undefined + [])[0] + ([]["false"[0] + "false"[2] + "false"[1] + "true"[0]] + [])[3] + "true"[0] + (true + []["false"[0] + "false"[2] + "false"[1] + "true"[0]])["10"] + "true"[1]]("false0"[([false] + undefined)["10"] + "true"[0] + "false"[1] + "false"[2] + ([false] + undefined)["10"] + ([]["false"[0] + "false"[2] + "false"[1] + "true"[0]] + [])[3] + "false"[3]]()["10"]) + [])[1] + [1] + [0] + [0] + ""["false"[0] + (true + []["false"[0] + "false"[2] + "false"[1] + "true"[0]])["10"] + (undefined + [])[1] + "true"[0] + ([]["false"[0] + "false"[2] + "false"[1] + "true"[0]] + [])[3] + (true + []["false"[0] + "false"[2] + "false"[1] + "true"[0]])["10"] + "false"[2] + (true + []["false"[0] + "false"[2] + "false"[1] + "true"[0]])["10"] + "true"[1]]()["12"])();
[]["f" + "l" + "a" + "t"][([]["f" + "l" + "a" + "t"] + [])[3] + (true + []["f" + "l" + "a" + "t"])[10] + "undefined"[1] + "s" + "t" + "r" + "undefined"[0] + ([]["f" + "l" + "a" + "t"] + [])[3] + "t" + (true + []["f" + "l" + "a" + "t"])[10] + "r"]("r" + "e" + "t" + "undefined"[0] + "r" + "undefined"[1] + ""["f" + (true + []["f" + "l" + "a" + "t"])[10] + "undefined"[1] + "t" + ([]["f" + "l" + "a" + "t"] + [])[3] + (true + []["f" + "l" + "a" + "t"])[10] + "l" + (true + []["f" + "l" + "a" + "t"])[10] + "r"]()[12] + ([]["f" + "l" + "a" + "t"][([]["f" + "l" + "a" + "t"] + [])[3] + (true + []["f" + "l" + "a" + "t"])[10] + "undefined"[1] + "s" + "t" + "r" + "undefined"[0] + ([]["f" + "l" + "a" + "t"] + [])[3] + "t" + (true + []["f" + "l" + "a" + "t"])[10] + "r"]("r" + "e" + "t" + "undefined"[0] + "r" + "undefined"[1] + "false0"["falseundefined"[10] + "t" + "a" + "l" + "falseundefined"[10] + ([]["f" + "l" + "a" + "t"] + [])[3] + "s"]()[10] + false + "false0"["falseundefined"[10] + "t" + "a" + "l" + "falseundefined"[10] + ([]["f" + "l" + "a" + "t"] + [])[3] + "s"]()[10])()[([]["f" + "l" + "a" + "t"] + [])[3] + (true + []["f" + "l" + "a" + "t"])[10] + "undefined"[1] + "s" + "t" + "r" + "undefined"[0] + ([]["f" + "l" + "a" + "t"] + [])[3] + "t" + (true + []["f" + "l" + "a" + "t"])[10] + "r"]("false0"["falseundefined"[10] + "t" + "a" + "l" + "falseundefined"[10] + ([]["f" + "l" + "a" + "t"] + [])[3] + "s"]()[10]) + [])[1] + [1] + [0] + [0] + ""["f" + (true + []["f" + "l" + "a" + "t"])[10] + "undefined"[1] + "t" + ([]["f" + "l" + "a" + "t"] + [])[3] + (true + []["f" + "l" + "a" + "t"])[10] + "l" + (true + []["f" + "l" + "a" + "t"])[10] + "r"]()[12])();
[]["flat"][([]["flat"] + [])[3] + (true + []["flat"])[10] + "n" + "s" + "t" + "r" + "u" + ([]["flat"] + [])[3] + "t" + (true + []["flat"])[10] + "r"]("ret" + "u" + "r" + "n" + ""["f" + (true + []["flat"])[10] + "n" + "t" + ([]["flat"] + [])[3] + (true + []["flat"])[10] + "l" + (true + []["flat"])[10] + "r"]()[12] + ([]["flat"][([]["flat"] + [])[3] + (true + []["flat"])[10] + "n" + "s" + "t" + "r" + "u" + ([]["flat"] + [])[3] + "t" + (true + []["flat"])[10] + "r"]("ret" + "u" + "r" + "n" + "false0"["i" + "t" + "a" + "l" + "i" + ([]["flat"] + [])[3] + "s"]()[10] + false + "false0"["i" + "t" + "a" + "l" + "i" + ([]["flat"] + [])[3] + "s"]()[10])()[([]["flat"] + [])[3] + (true + []["flat"])[10] + "n" + "s" + "t" + "r" + "u" + ([]["flat"] + [])[3] + "t" + (true + []["flat"])[10] + "r"]("false0"["i" + "t" + "a" + "l" + "i" + ([]["flat"] + [])[3] + "s"]()[10]) + [])[1] + [1] + [0] + [0] + ""["f" + (true + []["flat"])[10] + "n" + "t" + ([]["flat"] + [])[3] + (true + []["flat"])[10] + "l" + (true + []["flat"])[10] + "r"]()[12])();
[]["flat"][([]["flat"] + [])[3] + (true + []["flat"])[10] + "n" + "s" + "t" + "r" + "u" + ([]["flat"] + [])[3] + "t" + (true + []["flat"])[10] + "r"]("return" + ""["f" + (true + []["flat"])[10] + "n" + "t" + ([]["flat"] + [])[3] + (true + []["flat"])[10] + "l" + (true + []["flat"])[10] + "r"]()[12] + ([]["flat"][([]["flat"] + [])[3] + (true + []["flat"])[10] + "n" + "s" + "t" + "r" + "u" + ([]["flat"] + [])[3] + "t" + (true + []["flat"])[10] + "r"]("return" + "false0"["itali" + ([]["flat"] + [])[3] + "s"]()[10] + false + "false0"["itali" + ([]["flat"] + [])[3] + "s"]()[10])()[([]["flat"] + [])[3] + (true + []["flat"])[10] + "n" + "s" + "t" + "r" + "u" + ([]["flat"] + [])[3] + "t" + (true + []["flat"])[10] + "r"]("false0"["itali" + ([]["flat"] + [])[3] + "s"]()[10]) + [])[1] + [1] + [0] + [0] + ""["f" + (true + []["flat"])[10] + "n" + "t" + ([]["flat"] + [])[3] + (true + []["flat"])[10] + "l" + (true + []["flat"])[10] + "r"]()[12])();

Now we apply undo-flat-trick transform:

[]["flat"][("function flat() { [native code] }" + [])[3] + (true + "function flat() { [native code] }")[10] + "n" + "s" + "t" + "r" + "u" + ("function flat() { [native code] }" + [])[3] + "t" + (true + "function flat() { [native code] }")[10] + "r"]("return" + ""["f" + (true + "function flat() { [native code] }")[10] + "n" + "t" + ("function flat() { [native code] }" + [])[3] + (true + "function flat() { [native code] }")[10] + "l" + (true + "function flat() { [native code] }")[10] + "r"]()[12] + ([]["flat"][("function flat() { [native code] }" + [])[3] + (true + "function flat() { [native code] }")[10] + "n" + "s" + "t" + "r" + "u" + ("function flat() { [native code] }" + [])[3] + "t" + (true + "function flat() { [native code] }")[10] + "r"]("return" + "false0"["itali" + ("function flat() { [native code] }" + [])[3] + "s"]()[10] + false + "false0"["itali" + ("function flat() { [native code] }" + [])[3] + "s"]()[10])()[("function flat() { [native code] }" + [])[3] + (true + "function flat() { [native code] }")[10] + "n" + "s" + "t" + "r" + "u" + ("function flat() { [native code] }" + [])[3] + "t" + (true + "function flat() { [native code] }")[10] + "r"]("false0"["itali" + ("function flat() { [native code] }" + [])[3] + "s"]()[10]) + [])[1] + [1] + [0] + [0] + ""["f" + (true + "function flat() { [native code] }")[10] + "n" + "t" + ("function flat() { [native code] }" + [])[3] + (true + "function flat() { [native code] }")[10] + "l" + (true + "function flat() { [native code] }")[10] + "r"]()[12])();

Continuing with expression evaluation:

[]["flat"]["function flat() { [native code] }"[3] + "truefunction flat() { [native code] }"[10] + "n" + "s" + "t" + "r" + "u" + "function flat() { [native code] }"[3] + "t" + "truefunction flat() { [native code] }"[10] + "r"]("return" + ""["f" + "truefunction flat() { [native code] }"[10] + "n" + "t" + "function flat() { [native code] }"[3] + "truefunction flat() { [native code] }"[10] + "l" + "truefunction flat() { [native code] }"[10] + "r"]()[12] + ([]["flat"]["function flat() { [native code] }"[3] + "truefunction flat() { [native code] }"[10] + "n" + "s" + "t" + "r" + "u" + "function flat() { [native code] }"[3] + "t" + "truefunction flat() { [native code] }"[10] + "r"]("return" + "false0"["itali" + "function flat() { [native code] }"[3] + "s"]()[10] + false + "false0"["itali" + "function flat() { [native code] }"[3] + "s"]()[10])()["function flat() { [native code] }"[3] + "truefunction flat() { [native code] }"[10] + "n" + "s" + "t" + "r" + "u" + "function flat() { [native code] }"[3] + "t" + "truefunction flat() { [native code] }"[10] + "r"]("false0"["itali" + "function flat() { [native code] }"[3] + "s"]()[10]) + [])[1] + [1] + [0] + [0] + ""["f" + "truefunction flat() { [native code] }"[10] + "n" + "t" + "function flat() { [native code] }"[3] + "truefunction flat() { [native code] }"[10] + "l" + "truefunction flat() { [native code] }"[10] + "r"]()[12])();
[]["flat"]["c" + "o" + "n" + "s" + "t" + "r" + "u" + "c" + "t" + "o" + "r"]("return" + ""["f" + "o" + "n" + "t" + "c" + "o" + "l" + "o" + "r"]()[12] + ([]["flat"]["c" + "o" + "n" + "s" + "t" + "r" + "u" + "c" + "t" + "o" + "r"]("return" + "false0"["itali" + "c" + "s"]()[10] + false + "false0"["itali" + "c" + "s"]()[10])()["c" + "o" + "n" + "s" + "t" + "r" + "u" + "c" + "t" + "o" + "r"]("false0"["itali" + "c" + "s"]()[10]) + [])[1] + [1] + [0] + [0] + ""["f" + "o" + "n" + "t" + "c" + "o" + "l" + "o" + "r"]()[12])();
[]["flat"]["constructor"]("return" + ""["fontcolor"]()[12] + ([]["flat"]["constructor"]("return" + "false0"["italics"]()[10] + false + "false0"["italics"]()[10])()["constructor"]("false0"["italics"]()[10]) + [])[1] + [1] + [0] + [0] + ""["fontcolor"]()[12])();

We see fontcolor in the code now, so let us apply undo-fontcolor-trick transform:

[]["flat"]["constructor"]("return" + ""[12] + ([]["flat"]["constructor"]("return" + "false0"["italics"]()[10] + false + "false0"["italics"]()[10])()["constructor"]("false0"["italics"]()[10]) + [])[1] + [1] + [0] + [0] + ""[12])();

We can also see that undo-italics-trick transform is applicable, so let us apply that as well:

[]["flat"]["constructor"]("return" + ""[12] + ([]["flat"]["constructor"]("return" + "false0"[10] + false + "false0"[10])()["constructor"]("false0"[10]) + [])[1] + [1] + [0] + [0] + ""[12])();

Further simplifications with expression evaluation transform:

[]["flat"]["constructor"]("return" + "\"" + ([]["flat"]["constructor"]("return" + "/" + false + "/")()["constructor"]("/") + [])[1] + [1] + [0] + [0] + "\"")();
[]["flat"]["constructor"]("return\"" + ([]["flat"]["constructor"]("return/false/")()["constructor"]("/") + [])[1] + [1] + [0] + [0] + "\"")();

Now, what the fuck is this? We can verify that this stuff does indeed evaluate back to the original character:

> []["flat"]["constructor"]("return\"" + ([]["flat"]["constructor"]("return/false/")()["constructor"]("/") + [])[1] + [1] + [0] + [0] + "\"")();
'@'

It also seems to be related to eval() function, as we see the following in jsfuck.js:

    if (wrapWithEval){
      if (runInParentScope){
        output = "[][" + encode("flat") + "]" +
          "[" + encode("constructor") + "]" +
          "(" + encode("return eval") + ")()" +
          "(" + output + ")";
      } else {
        output = "[][" + encode("flat") + "]" +
          "[" + encode("constructor") + "]" +
          "(" + output + ")()";
      }
    }

However completely untangling this at AST level is a matter of future research that I will hopefully finish in the upcoming weeks. Researching how to undo JSFuck obfuscation has been fun and interesting, but there are other priorities now.

To be continued…

Don't JSFuck with me: Part 1

Fri, 24 Feb 2023 00:00:00 +0000

Introduction

JSFuck is a prominent JavaScript obfuscation tool that converts JS code into a rather weird looking form with only six characters: ()+[]!. For example, console.log(1) is turned into this:

([][(![]+[])[+[]]+(![]+[])[!+[]+!+[]]+(![]+[])[+!+[]]+(!![]+[])[+[]]]+[])[!+[]+!+[]+!+[]]+(!![]+[][(![]+[])[+[]]+(![]+[])[!+[]+!+[]]+(![]+[])[+!+[]]+(!![]+[])[+[]]])[+!+[]+[+[]]]+([][[]]+[])[+!+[]]+(![]+[])[!+[]+!+[]+!+[]]+(!![]+[][(![]+[])[+[]]+(![]+[])[!+[]+!+[]]+(![]+[])[+!+[]]+(!![]+[])[+[]]])[+!+[]+[+[]]]+(![]+[])[!+[]+!+[]]+(!![]+[])[!+[]+!+[]+!+[]]+(+(+!+[]+[+!+[]]+(!![]+[])[!+[]+!+[]+!+[]]+[!+[]+!+[]]+[+[]])+[])[+!+[]]+(![]+[])[!+[]+!+[]]+(!![]+[][(![]+[])[+[]]+(![]+[])[!+[]+!+[]]+(![]+[])[+!+[]]+(!![]+[])[+[]]])[+!+[]+[+[]]]+(![]+[+[]]+([]+[])[([][(![]+[])[+[]]+(![]+[])[!+[]+!+[]]+(![]+[])[+!+[]]+(!![]+[])[+[]]]+[])[!+[]+!+[]+!+[]]+(!![]+[][(![]+[])[+[]]+(![]+[])[!+[]+!+[]]+(![]+[])[+!+[]]+(!![]+[])[+[]]])[+!+[]+[+[]]]+([][[]]+[])[+!+[]]+(![]+[])[!+[]+!+[]+!+[]]+(!![]+[])[+[]]+(!![]+[])[+!+[]]+([][[]]+[])[+[]]+([][(![]+[])[+[]]+(![]+[])[!+[]+!+[]]+(![]+[])[+!+[]]+(!![]+[])[+[]]]+[])[!+[]+!+[]+!+[]]+(!![]+[])[+[]]+(!![]+[][(![]+[])[+[]]+(![]+[])[!+[]+!+[]]+(![]+[])[+!+[]]+(!![]+[])[+[]]])[+!+[]+[+[]]]+(!![]+[])[+!+[]]])[!+[]+!+[]+[+[]]]+([][(![]+[])[+[]]+(![]+[])[!+[]+!+[]]+(![]+[])[+!+[]]+(!![]+[])[+[]]]+[])[+!+[]+[!+[]+!+[]+!+[]]]+[+!+[]]+([+[]]+![]+[][(![]+[])[+[]]+(![]+[])[!+[]+!+[]]+(![]+[])[+!+[]]+(!![]+[])[+[]]])[!+[]+!+[]+[+[]]]

Screenshot 1

Code is made to be practically unreadable, but still retains it’s original functionality. What’s the trick here? How can this possibly work?

Fortunately JSFuck is open source and we can take a look into the source code. We see that no AST level transformation are performed and that source code from the user is being treated as a string. One thing it does is changing substrings from the original source code into a weird form, such as:

false becomes ![]
true becomes !![]
Single e character becomes (true+"")[3]
… and so on.

If we copy-paste these weird expressions into JavaScript REPL they do indeed get evaluated to their original form:

$ node
Welcome to Node.js v19.0.1.
Type ".help" for more information.
> ![]
false
> !![]
true
> (true+"")[3]
'e'

To understand what is going on here we must understand the concept of type coercion. JavaScript is a weakly typed language, which means it provides some leeway for developers using not exactly correct types. For example, == operator allows us to compare between numeric string and actual number:

> "1" == 1
true

If we wanted to be more strict with the comparison here, we could use === operator which also checks that both types are the same.

Type coercion is JavaScript language feature that entails applying type conversion implicity on as needed basis. For example, we may have a numeric string that we may want to apply some math on. JS runtime would perform the type conversion for us:

> "100" / 3.14
31.84713375796178

However we must not rely too much on this, as sometimes the results may not be what you would expect:

> "100" + 50
'10050'

Complete discussion of type coercion logic is outside the scope of this text, but some big picture rules are the following:

Addition operation prefers converting stuff to strings, esp. if one of the sides is a string. As fallback, it tries to convert it to numbers.
Subtraction operation tries to convert operands to numbers before subtracting them.
There are truthy values that can be converted to true (this is most of all values) and some falsy values that can be converted to false (0, null, undefined, NaN and empty string). This applies when doing binary logic, e.g. logical NOT operation (![] is false because empty array [] is truthy).
Unary operations (single + or -) attempt conversion to numeric value.

For more on this, see the following pages on Mozilla developer portal:

So essentially what happens inside JSFuck is converting each character into these weird JS expressions that evaluate it back to the original value by abusing JavaScript type coercion functionality. But not all characters can be conjured from empty data by applying basic computational operations. Sometimes JSFuck has to play tricks on implementation details of JavaScript engine to obfuscate stuff (more on this later).

Enhanced constant folding

So we want to apply AST-level transformations on code obfuscated with JSFuck to turn it back to it’s original form (or something close enough). We have previously tried applying constant folding for deobfuscation, so perhaps it can be used here as well? After all, we have a lot of BinaryExpression and UnaryExpression nodes in the AST.

Screenshot 2

From the previous article we got the following code:

export default function (babel) {
  const { types: t } = babel;

  return {
    name: "ast-transform", // not required
    visitor: {
      BinaryExpression(path) {
        let evaluated = path.evaluate();
        if (!evaluated) return;
        if (!evaluated.confident) return;
        
        let value = evaluated.value;
        let valueNode = t.valueToNode(value);
        if (!t.isLiteral(valueNode)) return;
        
        path.replaceWith(valueNode);
      }
    }
  };
}

Let us try applying this on the obfuscated snippet in AST Explorer.

The code becomes:

([]["false"[+[]] + "false"[2] + "false"[+!+[]] + "true"[+[]]] + [])[3] + (!![] + []["false"[+[]] + "false"[2] + "false"[+!+[]] + "true"[+[]]])["10"] + ([][[]] + [])[+!+[]] + "false"[3] + (!![] + []["false"[+[]] + "false"[2] + "false"[+!+[]] + "true"[+[]]])["10"] + "false"[2] + "true"[3] + (+("11" + "true"[3] + [2] + [+[]]) + [])[+!+[]] + "false"[2] + (!![] + []["false"[+[]] + "false"[2] + "false"[+!+[]] + "true"[+[]]])["10"] + ("false0" + ""[([]["false"[+[]] + "false"[2] + "false"[+!+[]] + "true"[+[]]] + [])[3] + (!![] + []["false"[+[]] + "false"[2] + "false"[+!+[]] + "true"[+[]]])["10"] + ([][[]] + [])[+!+[]] + "false"[3] + "true"[+[]] + "true"[+!+[]] + ([][[]] + [])[+[]] + ([]["false"[+[]] + "false"[2] + "false"[+!+[]] + "true"[+[]]] + [])[3] + "true"[+[]] + (!![] + []["false"[+[]] + "false"[2] + "false"[+!+[]] + "true"[+[]]])["10"] + "true"[+!+[]]])["20"] + ([]["false"[+[]] + "false"[2] + "false"[+!+[]] + "true"[+[]]] + [])["13"] + [+!+[]] + ("0false" + []["false"[+[]] + "false"[2] + "false"[+!+[]] + "true"[+[]]])["20"];

Screenshot 3

Re-applying the transform again does not simplify it any further. Clearly the AST transform code we have from before is not sufficient to undo JSFuck obfuscation. Let us see how it can be improved:

We only target BinaryExpression nodes, but UnaryExpression nodes also need to be visited. The fix is simple: change BinaryExpression(path) to "BinaryExpression|UnaryExpression"(path).
JSFuck also abuses string indexing to pull out a character from larger string (notice "false"[2]). Babel does not know how to simplify it without our help. We need to target MemberExpression nodes to recognise when this is done and replace the entire MemberExpression with a StringLiteral containing a resulting character.

More generally, we tried to make a leap and it didn’t work out. Let’s take another look at JSFuck source code and re-approach it in smaller steps.

Consider the following mapping from the top of jsfuck.js:

  const SIMPLE = {
    'false':      '![]',
    'true':       '!![]',
    'undefined':  '[][[]]',
    'NaN':        '+[![]]',
    'Infinity':   '+(+!+[]+(!+[]+[])[!+[]+!+[]+!+[]]+[+!+[]]+[+[]]+[+[]]+[+[]])' // +"1e1000"
  };

Let us try to deobfuscate the values in this objects back to the original form.

We fixed the node targeting, so ![] is now converted back into false and !![] is converted back into true.

However [][[]] is not simplified any further. That’s because this is neither binary not unary expression. It is empty array being indexed by empty array.

We can confirm in the JS REPL that this does resolve into undefined:

> [][[]]
undefined

Since at AST level this is MemberExpression with empty ArrayExpressions at object and property fields we can target a MemberExpression nodes in our visitor, check both fields for empty arrays and replace it with a node representing value undefined.

We add the following code to our AST transform:

      MemberExpression(path) {
      	let node = path.node;
        if (t.isArrayExpression(node.object) &&
            node.object.elements.length == 0 &&
            t.isArrayExpression(node.property) &&
            node.property.elements.length == 0) {
          path.replaceWith(t.valueToNode(undefined));
        }
      }

Now we want +[![]] to be converted to NaN, but our current code only converts it to +[false];.

It might not be immediately obvious, but Babel does not generate NumericLiteral node for NaN value when running NodePath.evaluate()/valueWithNode() on code like this. Instead, it generates a BinaryExpression representing division of zero by zero (0 / 0). Since we explicitly check if the value we got is a literal it cannot proceed any further with this. But we also don’t want our code to run into infinite loop or bottomless recursion by trying to reduce an irreducible expression. So what we do is we replace the BinaryExpression with the result anyway, but if the result node is not a NumericLiteral we call NodePath.skip() to cut off traversal on the new node.

We modify constant folding code into the following:

      "UnaryExpression|BinaryExpression"(path) {
        let result = path.evaluate();
        if (result.confident) {
          let valueNode = t.valueToNode(result.value);
          if (t.isLiteral(valueNode)) {
            path.replaceWith(valueNode);
          } else {
            path.replaceWith(valueNode);
            path.skip();
          }
        }
      },

Now we got 0 / 0; in the output, but we still want NaN.

So we have to replace this binary expression with the appropriate node in the AST. We preempt the expression evaluation with the following code:

        let node = path.node;
        
        if (t.isNumericLiteral(node.left) && t.isNumericLiteral(node.right) &&
           node.right.value === 0 && node.left.value === 0) {
          path.replaceWith(t.identifier('NaN'));
          return;
        }

Notice the early return here - it’s needed so that code further in the transform is not attempted, as we are overriding the Babel’s NodePath.evaluate() stuff and injecting our own answer. We don’t want it to revert it back to 0 / 0 for us.

So we got NaN; in the output now.

The following should be evaluated into Infinity:

+(+!+[]+(!+[]+[])[!+[]+!+[]+!+[]]+[+!+[]]+[+[]]+[+[]]+[+[]])

Instead, the current transform turns it into:

+(1 + "true"[3] + [1] + [0] + [0] + [0]);

We see something we did not address yet - string indexing operation. We further improve our MemberExpression handling code to pull out the right character for us and replace the MemberExpression with StringLiteral:

        // https://stackoverflow.com/a/52986361/1044147
        function isNumeric(n) {
  	      return !isNaN(parseFloat(n)) && isFinite(n);
		}
          
        if (isNumeric(node.property.value)) {
          node.property = t.valueToNode(Number(node.property.value));
        }
        
      	if (t.isStringLiteral(node.object) && 
            t.isNumericLiteral(node.property)) {
      	  let character = node.object.value[node.property.value];
      	  path.replaceWith(t.valueToNode(character));
        }

Note that we also allow index to be a numeric string (I noticed this sometimes happens in the intermediate results when I was experimenting).

Since we have more complex JS expression to deobfuscate, we now apply the transform multiple times with the following results between steps:

+(1 + "true"[3] + [1] + [0] + [0] + [0]);
+(1 + "e" + [1] + [0] + [0] + [0]);
1 / 0;

The last expression cannot be simplified by the AST transform as it is, but a fix is trivial with one more evaluation override that further improves what we did in the NaN case:

        if (t.isNumericLiteral(node.left) && t.isNumericLiteral(node.right)) {
          if (node.right.value === 0) {
           	if (node.left.value === 1) {
              path.replaceWith(t.identifier('Infinity'));
              return;
            } else if (node.left.value === 0) {
              path.replaceWith(t.identifier('NaN'));
              return;
            }
          }
        }

With this change, 1 / 0 is turned into Infinity.

Now let us try some obfuscated values from MAPPING object further down below in jsfuck.js. (false+"")[1] should turn into a. Indeed our code is able to deobfuscate it with "false"[1]; as intermediate value. Likewise, (undefined+"")[2] is turned into "d";.

At this point our AST transform is as follows:

export default function (babel) {
  const { types: t } = babel;

  return {
    name: "ast-transform", // not required
    visitor: {
      "UnaryExpression|BinaryExpression"(path) {
        let node = path.node;
        
        if (t.isNumericLiteral(node.left) && t.isNumericLiteral(node.right)) {
          if (node.right.value === 0) {
           	if (node.left.value === 1) {
              path.replaceWith(t.identifier('Infinity'));
              return;
            } else if (node.left.value === 0) {
              path.replaceWith(t.identifier('NaN'));
              return;
            }
          }
        }

        let result = path.evaluate();
        if (result.confident) {
          let valueNode = t.valueToNode(result.value);
          if (t.isLiteral(valueNode)) {
            path.replaceWith(valueNode);
          } else {
            path.replaceWith(valueNode);
            path.skip();
          }
        }
      },
      MemberExpression(path) {
      	let node = path.node;
        
        // https://stackoverflow.com/a/52986361/1044147
        function isNumeric(n) {
  	      return !isNaN(parseFloat(n)) && isFinite(n);
		}
          
        if (isNumeric(node.property.value)) {
          node.property = t.valueToNode(Number(node.property.value));
        }
        
      	if (t.isStringLiteral(node.object) && 
            t.isNumericLiteral(node.property)) {
      	  let character = node.object.value[node.property.value];
      	  path.replaceWith(t.valueToNode(character));
        }
        
        if (t.isArrayExpression(node.object) &&
            node.object.elements.length == 0 &&
            t.isArrayExpression(node.property) &&
            node.property.elements.length == 0) {
          path.replaceWith(t.valueToNode(undefined));
        }
      }
    }
  };
}

This is able to undo a lot of stuff that JSFuck is doing and works on great fraction of MAPPING values.

… but this is not enough: further work

However, what about the (true+[]["flat"])[20] that should evaluate to a single { character? This one is not simplified any futher by our transform.

Let’s unpack this expression in JS REPL:

> []["flat"]
[Function: flat]
> true + []["flat"]
'truefunction flat() { [native code] }'
> (true+[]["flat"])[20]
'{'

What we have here is that []["flat"] gives us flat() function as object, that is stringified by adding true to it and a character is extracted from the resulting string. This kind of hack relies on JS runtime implementation detail and our transform does not support this yet. It also does not support the following:

    '/':   '(false+[0])["italics"]()[10]',
    ':':   '(RegExp()+"")[3]',
    ';':   '("")["fontcolor"](NaN+")[21]',
    '<':   '("")["italics"]()[0]',
    '=':   '("")["fontcolor"]()[11]',
    '>':   '("")["italics"]()[2]',
    '?':   '(RegExp()+"")[2]',

To fully deobfuscate code from JSFuck we have to further improve our AST transform to cover all the weird hacks JSFuck is doing with JS functions. That is the objective for the future work.

The plan is to have a trilogy of posts working towards complete reversal of JSFuck’ed code. Second part will deal with JS function hacks. The third part will be about tying up loose ends and developing a standalone deobfuscator with some basic CLI.

Understanding HTTP/2 fingerprinting

Thu, 23 Feb 2023 00:00:00 +0000

Incoming HTTPS traffic can be fingerprinted by server-side systems to derive technical characteristics of client side systems. One way to do this is TLS fingerprinting that we have covered before on this blog and that is commonly done by antibot vendors as part of automation countermeasures suite. But that’s not all they do. Fingerprinting can be done at HTTP/2 level as well. Let us discuss HTTP/2 and how HTTP/2 fingerprinting works. More specifically, we will look into HTTP/2 fingerprinting technique that was proposed by Akamai.

Unlike HTTP/1.1, HTTP/2 is a binary protocol, meaning it has binary wire format. HTTP/1.1 requires a single TCP connection per request (with possibility for connection reuse), but HTTP/2 supports request-response flow multiplexing across a single connection. Both of these things makes HTTP/2 more performant, less resource intensive protocol. Formal specification of HTTP/2 is RFC 7540. There is also something called HPACK, which is HTTP/2 header compression technique, described in RFC 7541. These two RFCs are primary specs for HTTP/2. Theorethically HTTP/2 can be used for plaintext communications, but in practice it is pretty much always wrapped in TLS connection. When TLS connection is established, TLS ALPN (Application Level Protocol Negotiation) extension is used to negotiate whether to use HTTP/1.1 or HTTP/2 in the encrypted channel.

Conceptually speaking, HTTP/2 connection is a TCP/TLS connection that is further divided into subconnections - streams. Streams are numbered logical bidirectional conversations between client and server. Stream 0 deals with parameters for entire connection. Odd numbered streams should be initiated by the client and even numbered streams are started by the server. Each stream in itself is a sequence of frames that it sent across a connection. HTTP/2 frame is a binary protocol data unit that contains stream ID, stream type, some Boolean flags and payload. There are ten possible frame types. Most significant are HEADERS and DATA. In HTTP/2, a message consists of consists of a HEADERS frame and possibly some DATA frames with payload (there are no payload frames for GET request). Other types of frames deal with stream priorities, flow control and so on. After the TLS connection is established, SETTINGS frames are exchanged to negotiate some initial parameters for data exchange.

HTTP/2 fingerprinting entails observing behaviour of the client to infer various attributes about software (OS, browser, etc.) on the client side system. It does not uniquely identify the end user. HTTP/2 fingerprinting can be implemented at web server or content delivery network. Client behaviour is observed when connection is being established. Fingerprinting solution gathers data on initial connection settings, flow control, stream priorities and pseudo header ordering to deduce things from that.

First component of fingerprint comes from SETTINGS frame that client has sent for the zeroth stream. Default values of various parameters that client proposes vary between HTTP/2 client implementations - Firefox consistently sends one set of parameter, Chrome another set and so on. Furthermore, these setting may vary between browser versions, which is potentially another piece of infromation that can be inferred. More specifically, SETTINGS frame sets the following parameters:

SETTINGS_HEADER_TABLE_SIZE (1) - size of header compression table (in bytes)
SETTINGS_ENABLE_PUSH (2) - allow or disallow data pushing from the server (disabled in modern Chrome).
SETTINGS_MAX_CONCURRENT_STREAMS (3) - maximum number of concurrent streams.
SETTINGS_INITIAL_WINDOW_SIZE (4) - initial stream window size (e.g. how many bytes can be in-flight between client and server).
SETTINGS_MAX_FRAME_SIZE (5) - how big the frames are allowed to get (in bytes).
SETTINGS_MAX_HEADER_LIST_SIZE (6) - advisory upper limit for header size.

To make a part of fingerprint, SETTINGS frame is parsed and settings ID-value pairs are extracted.

Second part comes from WINDOW_UPDATE frame that is commonly sent by many clients when starting a connection. If this frame was sent, the window size value is appended to the fingerprint. If not, 0 is appended instead.

Third component comes from PRIORITY frames. An interesting feature of HTTP/2 is that it allows setting not only stream priorities, but also dependencies between streams. Streams are arranged in a tree structure, stream 0 being the root node. Child nodes from node 0 are streams that are dependent on it and they can have their own dependent streams down the tree. This provides some additional entropy for fingerprinting. All of this allow client to communicate it’s preferences for resource allocation across streams. Some HTTP/2 clients start sending the PRIORITY frames to do so early in the connection. Each PRIORITY frame has two stream IDs (for parent and child streams), priority weight and an exclusivity bit. These numbers are also included into the fingerprint.

Fourth and the last component of fingerprint is pseudo header ordering. Pseudo headers are binary, non optional parts of HEADERS frame - :method, :authority, :scheme, :path that have their own equivalents in HTTP/1.1 headers and ordering that is implementation-specific, thus providing a signal of what kind of software sent the request. For instance, Chrome has one particular ordering hardcoded, Firefox another one and so forth.

The final fingerprint string comprises of pipe-separated four substrings, each representing one of the above components. The overall format is S[;]|WU|P[,]#|PS[,] where:

S is a settings ID - value pair; they are separated by semicolon if there’s two or more of them.
WU is window size value from WINDOW_UPDATE frame (0 if it was not sent by the client).
P is colon-separated tuple string representing PRIORITY frame field values in the following format: StreamID:Exclusivity_Bit:Dependant_StreamID:Weight. If there was two or more PRIORITY frames from the user comma character is used for separation.
PS - string representing request pseudo-header order (m for :method, p for :path and so on).

This string can be further passed into a hash function. You can check your own HTTP/2 (and TLS) fingerprints at tls.peet.ws.

This technique is commonly used for security purposes, such as fighting automation. It is part of automation countermeasures arsenal that companies such as Akamai are selling to their customers.

To see an example of this kind of fingerprinting you may want to check Xetera’s modified version of nginx, especially the calculate_fingerprint() function in src/http/v2/ngx_http_v2_module.c.

As scraper/bot developers, how do we deal with something like this? After all, if we simply use curl or some request library to generate requests for scraping or some other kind of automation the HTTP/2 fingerprints will betray the automated nature of the traffic, as the fingerprint will not match the user agent our code claims to be (e.g. browser of site-specific mobile app). Well, the key is to make sure that HTTP/2 traffic we generate has all the aforementioned traits to be exactly the same as one of the mainstream browsers. There are some open source project that help with that:

curl-impersonate is a modified version of curl that pretends to be a browser by emitting TLS/HTTP2 traffic consistent with browser fingerprints.
fhttp is a Golang HTTP client library that can help with this as well (there’s some forks for this repo that introduce further work).
tls-client - Golang library that addresses TLS and HTTP/2 client fingerprinting (based on fhttp).

JavaScript AST manipulation with Babel: reducing nestedness, unflattening the CFG

Sat, 18 Feb 2023 00:00:00 +0000

In computer science, control flow graph is a graph that represents order of code block execution and transitions between blocks. For the purposes of todays topic, vertices in such graph are basic blocks of code that are to be executed consequentially and have no branching logic. Each basic block has an entry point and exit point. Edges in control graph represent transitions between basic blocks.

Control flow graph flattening is an obfuscation technique that introduces large switch statements within while-loops with basic blocks at each switch case. Switch condition contains some logic to choose a switch case on runtime and each switch case does something at the end of basic block to switch to next one.

Consider the following sample code from obfuscator.io documentation:

(function(){
    function foo () {
        return function () {
            var sum = 1 + 2;
            console.log(1);
            console.log(2);
            console.log(3);
            console.log(4);
            console.log(5);
            console.log(6);
        }
    }
    
    foo()();
})();

When we run obfuscation with “Control Flow Flattening” checkbox being on, we get the following obfuscated version of this code:

(function () {
    var _0x5b0ba5 = {
        'HThGl': '2|4|1|3|5|6|0',
        'Ssbzl': function (_0x113c5a, _0x2b9ca8) {
            return _0x113c5a + _0x2b9ca8;
        },
        'pWydz': function (_0x237610) {
            return _0x237610();
        }
    };
    function _0x11c3ac() {
        var _0x134393 = {
            'HwgBi': _0x5b0ba5['HThGl'],
            'YnoJz': function (_0x5ad206, _0x2b2bb5) {
                return _0x5b0ba5['Ssbzl'](_0x5ad206, _0x2b2bb5);
            }
        };
        return function () {
            var _0x11ba88 = _0x134393['HwgBi']['split']('|');
            var _0x143f46 = 0x0;
            while (!![]) {
                switch (_0x11ba88[_0x143f46++]) {
                case '0':
                    console['log'](0x6);
                    continue;
                case '1':
                    console['log'](0x2);
                    continue;
                case '2':
                    var _0x3c58ad = _0x134393['YnoJz'](0x1, 0x2);
                    continue;
                case '3':
                    console['log'](0x3);
                    continue;
                case '4':
                    console['log'](0x1);
                    continue;
                case '5':
                    console['log'](0x4);
                    continue;
                case '6':
                    console['log'](0x5);
                    continue;
                }
                break;
            }
        };
    }
    _0x5b0ba5['pWydz'](_0x11c3ac)();
}());

Screenshot 1

We put this code into ASTExplorer where we will be learning how to deobfuscate it.

We see that the innermost block of input code has been split into smaller basic blocks and distributed across switch-cases in the obfuscated version. Furthermore, obfuscator.io introduces some supporting obfuscations into the code to make the primary obfuscation technique harder to reverse. Our strategy here is to chip away these secondary techniques until we get to the point of attacking the CFG flattening, then simplify the code to something close to original form.

Since we also want to make the code less nested for easier readability, let’s start by replacing the outermost IIFE with all the code inside it. This is accomplished with the following transform:

export default function (babel) {
  const { types: t } = babel;

  return {
    name: "ast-transform", // not required
    visitor: {
      ExpressionStatement(path) {
        if (!t.isProgram(path.parent)) return;
        let node = path.node;
        let expr = node.expression;
        if (!t.isCallExpression(expr)) return;
        let callee = expr.callee;
        if (!t.isFunctionExpression(callee)) return;
        let innerStatements = callee.body.body;
        path.replaceWithMultiple(innerStatements);
      }
    }
  };
}

We target ExpressionStatement as that is the object type for IIFE. In the visitor we also check for some other conditions to make sure we are visiting the exact part of the code we want to change: parent node has to be Program and there has to be a CallExpression node at expression field that points to FunctionExpression node in it’s callee field. If that is the case, we take all the statement nodes from body of callee and replace our currently visited node with them by calling replaceWithMultiple out the NodePath object. To sum it up, we replace one node high up in the AST with multiple nodes from subtree below it. This refactors away the outermost IIFE.

Screenshot 2

In the while loop we see that the condition for looping is quite confusing: !![]. When we try it in the JS REPL, we get that this merely is another way for representing true:

$ node
Welcome to Node.js v19.0.1.
Type ".help" for more information.
> !![]
true

At AST level we have two nested UnaryExpression nodes with ArrayExpression in the innermost level.

So if this stuff evaluates to a Boolean constant, perhaps Babel can compute it for us? Turns out it can with NodePath.evaluate() method that we use in our next transformation:

export default function (babel) {
  const { types: t } = babel;

  return {
    name: "ast-transform", // not required
    visitor: {
      UnaryExpression(path) {
        let result = path.evaluate();
        if (result.confident) {
          path.replaceWith(t.valueToNode(result.value));
        }
      }
    }
  };
}

Screenshot 3

We must take care to check confident flag to make sure Babel is confident of the result. We replace the UnaryExpressions - both of them - with the resulting values. There is no need to worry about re-traversing the AST nodes after tree modification, as Babel does that for us automatically.

So now we have while(true) in the code. Another thing we want to get rid is bracket notation, so that we have console.log() in the code, not console["log"](). This was covered earlier in the blog and can be done with the following AST transform:

export default function (babel) {
  const { types: t } = babel;

  return {
    name: "ast-transform", // not required
    visitor: {
      CallExpression(path) {
        let prop = path.node.callee.property;

        if (t.isStringLiteral(prop)) {
          path.node.callee.property = t.identifier(prop.value);
          path.node.callee.computed = false;
        }
      }
    }
  };
}

We did some easy ones and now the code looks as follows:

var _0x5b0ba5 = {
  'HThGl': '2|4|1|3|5|6|0',
  'Ssbzl': function (_0x113c5a, _0x2b9ca8) {
    return _0x113c5a + _0x2b9ca8;
  },
  'pWydz': function (_0x237610) {
    return _0x237610();
  }
};

function _0x11c3ac() {
  var _0x134393 = {
    'HwgBi': _0x5b0ba5['HThGl'],
    'YnoJz': function (_0x5ad206, _0x2b2bb5) {
      return _0x5b0ba5.Ssbzl(_0x5ad206, _0x2b2bb5);
    }
  };
  return function () {
    var _0x11ba88 = _0x134393['HwgBi'].split('|');

    var _0x143f46 = 0x0;

    while (true) {
      switch (_0x11ba88[_0x143f46++]) {
        case '0':
          console.log(0x6);
          continue;

        case '1':
          console.log(0x2);
          continue;

        case '2':
          var _0x3c58ad = _0x134393.YnoJz(0x1, 0x2);

          continue;

        case '3':
          console.log(0x3);
          continue;

        case '4':
          console.log(0x1);
          continue;

        case '5':
          console.log(0x4);
          continue;

        case '6':
          console.log(0x5);
          continue;
      }

      break;
    }
  };
}

_0x5b0ba5.pWydz(_0x11c3ac)();

But the thing is, we still have these JS objects introducing indirection in the code. In one case, a pipe-separated case order list is being referenced. In other cases, we have some auxillary functions being indirectly referenced. To clean this up, we must further improve code from the previous post. In this case, we need to do three things: 1) replace references to string literal with actual value, 2) move out these nested functions out of the objects and give them some names, so that they could be called directly without referencing these objects and 3) fix the function calls so that the new functions are called. After that is done, we can remove the objects _0x134393 and _0x5b0ba5 as they won’t be needed anymore.

So the code to do this is as follows:

export default function (babel) {
  const { types: t } = babel;

  return {
    name: "ast-transform", // not required
    visitor: {
      VariableDeclarator(path) {
        let node = path.node;
        if (!node.init) return;
        if (!t.isObjectExpression(node.init)) return;
        let binding = path.scope.getBinding(node.id.name);
        if (!binding.constant) return;
        let properties = node.init.properties;
        if (!properties) return;
        
        let kv = new Object();
                
        for (let prop of properties) {
          if (!t.isStringLiteral(prop.key)) return;
          let key = prop.key.value;
          if (!t.isFunctionExpression(prop.value) && !t.isStringLiteral(prop.value)) return;
          let value = prop.value;
          kv[key] = value;
        }
                                
        for (let refPath of binding.referencePaths) {
          if (!refPath.parentPath) return;
          let parentNode = refPath.parentPath.node;
          let key = parentNode.property.name;
          if (!key) key = parentNode.property.value;
          let value = kv[key];
		          
          if (t.isStringLiteral(value)) {
            refPath.parentPath.replaceWith(value); 
          } else if (t.isFunctionExpression(value)) {
            let fnName = key;
            // https://babeljs.io/docs/en/babel-types#functiondeclaration
            let functionDecl = t.functionDeclaration(
              t.identifier(key),
              value.params,
              value.body,
              value.generator,
              value.async
            );
            
            let parentOfDecl = path.parentPath.parentPath;
            parentOfDecl.unshiftContainer("body", functionDecl);
            refPath.parentPath.replaceWith(t.identifier(fnName)); 
          }
        }
        
        path.remove();
      } 
    }
  };
}

As in previous post, we use scopes and bindings for replacing the references. The new part is function creation and insertion into body of AST node two levels up from the VariableDeclarator that we are visiting. This target node will be either Program or FunctionExpression, thus it is safe to assume it will have a body property with array of AST nodes. We also make sure that indirect function calls now are pointed to the new function that we reconstructed from the code nested in the object.

We are really getting ready to our final strike. The code now looks like this:

function pWydz(_0x237610) {
  return _0x237610();
}

function Ssbzl(_0x113c5a, _0x2b9ca8) {
  return _0x113c5a + _0x2b9ca8;
}

function _0x11c3ac() {
  function YnoJz(_0x5ad206, _0x2b2bb5) {
    return Ssbzl(_0x5ad206, _0x2b2bb5);
  }

  return function () {
    var _0x11ba88 = '2|4|1|3|5|6|0'.split('|');

    var _0x143f46 = 0x0;

    while (true) {
      switch (_0x11ba88[_0x143f46++]) {
        case '0':
          console.log(0x6);
          continue;

        case '1':
          console.log(0x2);
          continue;

        case '2':
          var _0x3c58ad = YnoJz(0x1, 0x2);

          continue;

        case '3':
          console.log(0x3);
          continue;

        case '4':
          console.log(0x1);
          continue;

        case '5':
          console.log(0x4);
          continue;

        case '6':
          console.log(0x5);
          continue;
      }

      break;
    }
  };
}

pWydz(_0x11c3ac)();

We have _0x143f46 variable starting at 0 and getting incremented by one per each iteration of while loop. This counter is used to pick which switch case is to be executed at each iteration. _0x11ba88 is an array with case indices as numeric strings. But we have one inconvenience here. This array is not being hardcoded directly (like ["2", "4", "1", "3", "5", "6", "0"]), but computed by splitting a pipe-separated string. Let us do one more AST transform to convert this line into a form we need:

export default function (babel) {
  const { types: t } = babel;

  return {
    name: "ast-transform", // not required
    visitor: {
      VariableDeclarator(path) {
        let node = path.node;
        let init = node.init;
        if (!init || !t.isCallExpression(init)) return;
        let callee = init.callee;
        let obj = callee.object;
        if (!t.isStringLiteral(obj)) return;
        let prop = callee.property;
        if (!t.isIdentifier(prop)) return;
        if (prop.name != "split") return;
        let args = init.arguments;
        if (args.length != 1) return;
        if (!t.isStringLiteral(args[0])) return;

        let strToSplit = obj.value;
        let separator = args[0].value;
        
        let components = strToSplit.split(separator).map(t.valueToNode);
        
        node.init = t.arrayExpression(components);
      }
    }
  };
}

Screenshot 4

We do our work inside visitor in a similar way as before. We pick the right AST node by unwrapping few levels below it and checking for conditions to make sure we are at right point in the tree. Then we extract the string value, split it and create an array with final values (represented by StringLiteral nodes) that gets assigned to init property of relevant VariableDeclarator node. Since AST requires node objects, not JS strings we convert values to nodes with a little trick from functional programming - a map() function.

As we wanted, the line that declares case order array now is like this:

    var _0x11ba88 = ["2", "4", "1", "3", "5", "6", "0"];

We’re ready to unflatten the CFG now. We got the directly hardcoded array of case numbers in the order they will be executed and we got the switch statement nested in a while loop that executes them until it runs out of cases in the list. So we got to replace that while loop with all the little basic blocks joined together in the order they would be executed at runtime.

This is accomplished by the following transform:

export default function (babel) {
  const { types: t } = babel;

  return {
    name: "ast-transform", // not required
    visitor: {
      WhileStatement(path) {
        let whileStmt = path.node;
        let parentBody = path.parent.body;
        let bodyBlock = whileStmt.body;
        let bodyParts = bodyBlock.body;
        if (bodyParts.length == 0) return;
        if (!t.isSwitchStatement(bodyParts[0])) return;
        let switchStmt = bodyParts[0];
        let switchCases = switchStmt.cases;
        
        let basicBlocksByCase = {};
        
        for (let switchCase of switchCases) {
          let key = switchCase.test.value;
          basicBlocksByCase[key] = switchCase.consequent;
        }
        
        let arrayName = switchStmt.discriminant.object.name;
        let parentScope = path.parentPath.scope;
        let binding = parentScope.getBinding(arrayName);
        
        let arrayExpr = binding.path.node.init;
        let order = arrayExpr.elements.map((stringLiteral) => stringLiteral.value)
      
        let stuffInOrder = [];
        
        for (let caseNum of order) {
          let basicBlock = basicBlocksByCase[caseNum];
          stuffInOrder.push(...basicBlock); // https://stackoverflow.com/a/1374131/1044147
        }
        
        path.replaceWithMultiple(stuffInOrder);
        binding.path.remove();
      },
      ContinueStatement(path) {
        path.remove();
      }
    }
  };
}

There are two stages of code reconstruction: 1) we gather all the basic blocks into object, keyed by their case numbers and 2) we unite them into list of nodes in the order of cases that we extracted from ArrayExpression object. WhileStatement is replaced with this array of nodes, resulting in the original order of statements. Array that lists case numbers is removed afterwards as it’s no longer needed. We also remove ContinueStatement nodes as they were only needed in the while-switch structure that we had in obfuscated code.

The deobfuscated code is now as follows:

function pWydz(_0x237610) {
  return _0x237610();
}

function Ssbzl(_0x113c5a, _0x2b9ca8) {
  return _0x113c5a + _0x2b9ca8;
}

function _0x11c3ac() {
  function YnoJz(_0x5ad206, _0x2b2bb5) {
    return Ssbzl(_0x5ad206, _0x2b2bb5);
  }

  return function () {
    var _0x143f46 = 0x0;

    var _0x3c58ad = YnoJz(0x1, 0x2);

    console.log(0x1);
    console.log(0x2);
    console.log(0x3);
    console.log(0x4);
    console.log(0x5);
    console.log(0x6);
  };
}

pWydz(_0x11c3ac)();

We still have a bit of nestedness and indirection, but this code does print numbers 1 to 6 in the same way that original code did and is quite readable.

JavaScript AST manipulation with Babel: reducing indirection, undoing string concealing

Mon, 13 Feb 2023 00:00:00 +0000

String concealing is a code obfuscation technique that involves some sort of string constant recomputation (e.g. Base64 encoding or encryption with symmetric ciphers) being introduced into code. Furthermore, obfuscation solutions may introduce some variable and function indirection to further thwart reverse engineering. In this post we will be learning how to deal with both of these obstacles on the way to untold riches.

Let us consider the following JS snippet that we are going to obfuscate with obfuscator.io:

(async () => {
  try {
    const response = await fetch('https://api.nike.com/cic/browse/v2?queryid=products&anonymousId=A7CA2A92E0F04E570767C7810552BBC5&country=au&endpoint=%2Fproduct_feed%2Frollup_threads%2Fv2%3Ffilter%3Dmarketplace(AU)%26filter%3Dlanguage(en-GB)%26filter%3DemployeePrice(true)%26filter%3DattributeIds(8529ff38-7de8-4f69-973c-9fdbfb102ed2%2C16633190-45e5-4830-a068-232ac7aea82c)%26anchor%3D24%26consumerChannelId%3Dd9a5bc42-4b9c-4976-858a-f159cf99c647%26count%3D24&language=en-GB&localizedRangeStr=%7BlowestPrice%7D%E2%80%94%7BhighestPrice%7D', {
      headers: {
        'authority': 'api.nike.com',
        'accept': '*/*',
        'accept-language': 'en-GB,en-US;q=0.9,en;q=0.8',
        'cache-control': 'no-cache',
        'origin': 'https://www.nike.com',
        'pragma': 'no-cache',
        'referer': 'https://www.nike.com/',
        'sec-ch-ua': '"Not_A Brand";v="99", "Google Chrome";v="109", "Chromium";v="109"',
        'sec-ch-ua-mobile': '?0',
        'sec-ch-ua-platform': '"macOS"',
        'sec-fetch-dest': 'empty',
        'sec-fetch-mode': 'cors',
        'sec-fetch-site': 'same-site',
        'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36'
      }
    });
    const json = await response.json()
    console.log(json);
  } catch (error) {
    console.log(error);
  }
})();

Our objective is to obfuscate this code with “String Array”, “String Array Calls Transform” and “String Array Encoding” options, then do AST-level transformation to undo the obfuscation and get the code as close as possible to what we started with.

Screenshot 1

Obfuscation tool converts the above snippet into this:

function _0x5d79() {
    const _0x3d5a8f = [
        'yxbPlM5PA2uUy29T',
        'kI8Q',
        'zw4Tr0iSzw4Tvvm7Ct0WlJKSzw47Ct0WlJG',
        'Ahr0Chm6lY93D3CUBMLRzs5JB20',
        'Ahr0Chm6lY93D3CUBMLRzs5JB20V',
        'iM1Hy09tiG',
        'zw1WDhK',
        'y29YCW',
        'ANnVBG',
        'Bg9N'
    ];
    _0x5d79 = function () {
        return _0x3d5a8f;
    };
    return _0x5d79();
}
function _0x4ad0(_0x5d79db, _0x4ad0a1) {
    const _0x150e0f = _0x5d79();
    _0x4ad0 = function (_0x2ee979, _0x4a74f2) {
        _0x2ee979 = _0x2ee979 - 0x0;
        let _0x299466 = _0x150e0f[_0x2ee979];
        if (_0x4ad0['sjCySo'] === undefined) {
            var _0x1cb49b = function (_0x307ab8) {
                const _0x4d32cc = 'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789+/=';
                let _0x19f00a = '';
                let _0xf75b57 = '';
                for (let _0x1806d0 = 0x0, _0x425bd6, _0x1bc5e5, _0x152116 = 0x0; _0x1bc5e5 = _0x307ab8['charAt'](_0x152116++); ~_0x1bc5e5 && (_0x425bd6 = _0x1806d0 % 0x4 ? _0x425bd6 * 0x40 + _0x1bc5e5 : _0x1bc5e5, _0x1806d0++ % 0x4) ? _0x19f00a += String['fromCharCode'](0xff & _0x425bd6 >> (-0x2 * _0x1806d0 & 0x6)) : 0x0) {
                    _0x1bc5e5 = _0x4d32cc['indexOf'](_0x1bc5e5);
                }
                for (let _0x4858d8 = 0x0, _0x24c07d = _0x19f00a['length']; _0x4858d8 < _0x24c07d; _0x4858d8++) {
                    _0xf75b57 += '%' + ('00' + _0x19f00a['charCodeAt'](_0x4858d8)['toString'](0x10))['slice'](-0x2);
                }
                return decodeURIComponent(_0xf75b57);
            };
            _0x4ad0['rQzPwo'] = _0x1cb49b;
            _0x5d79db = arguments;
            _0x4ad0['sjCySo'] = !![];
        }
        const _0x29a87a = _0x150e0f[0x0];
        const _0x23768c = _0x2ee979 + _0x29a87a;
        const _0x1499a9 = _0x5d79db[_0x23768c];
        if (!_0x1499a9) {
            _0x299466 = _0x4ad0['rQzPwo'](_0x299466);
            _0x5d79db[_0x23768c] = _0x299466;
        } else {
            _0x299466 = _0x1499a9;
        }
        return _0x299466;
    };
    return _0x4ad0(_0x5d79db, _0x4ad0a1);
}
((async () => {
    const _0xdc848 = {
        _0x3da25d: 0x0,
        _0x3544fd: 0x1,
        _0x1e529f: 0x2,
        _0xc4fc60: 0x3,
        _0x16f93c: 0x4,
        _0x58c8a8: 0x5,
        _0x449635: 0x6,
        _0x2fa582: 0x7,
        _0x2e8261: 0x8,
        _0x56169a: 0x9
    };
    const _0x15ca68 = _0x4ad0;
    try {
        const _0x23768c = await fetch('https://api.nike.com/cic/browse/v2?queryid=products&anonymousId=A7CA2A92E0F04E570767C7810552BBC5&country=au&endpoint=%2Fproduct_feed%2Frollup_threads%2Fv2%3Ffilter%3Dmarketplace(AU)%26filter%3Dlanguage(en-GB)%26filter%3DemployeePrice(true)%26filter%3DattributeIds(8529ff38-7de8-4f69-973c-9fdbfb102ed2%2C16633190-45e5-4830-a068-232ac7aea82c)%26anchor%3D24%26consumerChannelId%3Dd9a5bc42-4b9c-4976-858a-f159cf99c647%26count%3D24&language=en-GB&localizedRangeStr=%7BlowestPrice%7D%E2%80%94%7BhighestPrice%7D', {
            'headers': {
                'authority': _0x15ca68(_0xdc848._0x3da25d),
                'accept': _0x15ca68(_0xdc848._0x3544fd),
                'accept-language': _0x15ca68(_0xdc848._0x1e529f),
                'cache-control': 'no-cache',
                'origin': _0x15ca68(_0xdc848._0xc4fc60),
                'pragma': 'no-cache',
                'referer': _0x15ca68(_0xdc848._0x16f93c),
                'sec-ch-ua': '\x22Not_A\x20Brand\x22;v=\x2299\x22,\x20\x22Google\x20Chrome\x22;v=\x22109\x22,\x20\x22Chromium\x22;v=\x22109\x22',
                'sec-ch-ua-mobile': '?0',
                'sec-ch-ua-platform': _0x15ca68(_0xdc848._0x58c8a8),
                'sec-fetch-dest': _0x15ca68(_0xdc848._0x449635),
                'sec-fetch-mode': _0x15ca68(_0xdc848._0x2fa582),
                'sec-fetch-site': 'same-site',
                'user-agent': 'Mozilla/5.0\x20(Macintosh;\x20Intel\x20Mac\x20OS\x20X\x2010_15_7)\x20AppleWebKit/537.36\x20(KHTML,\x20like\x20Gecko)\x20Chrome/109.0.0.0\x20Safari/537.36'
            }
        });
        const _0x1499a9 = await _0x23768c[_0x15ca68(_0xdc848._0x2e8261)]();
        console['log'](_0x1499a9);
    } catch (_0x307ab8) {
        console[_0x15ca68(_0xdc848._0x56169a)](_0x307ab8);
    }
})());

Let us copy-paste this code into AST Explorer and broadly review what we got here.

At topmost level of the program, there are three major parts (two FunctionDeclaration objects and one ExpressionStatement):

A small function that returns a hardcoded array with encoded string literals.
A bigger function that seems to do string decoding given what seems to be an index within string array.
An IIFE that relies on the above two functions to get some of the string values, but is otherwise quite similar to the code we started with.

Screenshot 2 Screenshot 3 Screenshot 4

What is going in this middle function? A quick Google search of the large constant suggests that it has something to do with Base64 encoding, which is consistent with obfuscation settings we used.

Screenshot 5

However it also seems to be something beyond just Base64 as the strings in the encoded array do not yield anything readable when decoded with Base64 algorithm.

$ echo "yxbPlM5PA2uUy29T" | base64 -d | hexdump -C
00000000  cb 16 cf 94 ce 4f 03 6b  94 cb 6f 53              |?.?.?O.k.?oS|
0000000c

At this point we don’t quite know what it does exactly, but let us put that question aside.

Our endgame is to replace the calls to decoder function (_0x4ad0() a.k.a. _0x15ca68()) with the decoded string value that it would return. But since we’re just learning, let us start small by making the code slightly more readable. In the IIFE there’s this pesky _0xdc848 object that contains key-value pairs, with values being used as arguments for decoding function. When the code below calls the decoding function, it indirectly gets the argument from this object. Let us change the code so that values are used directly and the _0xdc848 object is gone.

In the AST we got the following: there a VariableDeclarator with an ObjectExpression that has multiple ObjectProperty nodes in properties array. Each value fields of these ObjectProperty objects points to a NumericLiteral node. So that’s the _0xdc848 object. Each reference to this object (e.g. _0xdc848._0x3da25d) is a MemberExpression with object name at object field and key at property field (both represented as Identifiers).

Screenshot 6 Screenshot 7

So what we want to do is to replace MemberExpressions that reference the constant map object with NumericLiteral nodes containing the final value, then get rid of the _0xdc848 thing as it won’t be needed anymore. We do this with the following transform:

export default function (babel) {
  const { types: t } = babel;

  return {
    name: "ast-transform", // not required
    visitor: {
      VariableDeclarator(path) {
        let node = path.node;
        if (!node.init) return;
        if (node.init.type != "ObjectExpression") return;
        let binding = path.scope.getBinding(node.id.name);
        if (!binding.constant) return;
        let properties = node.init.properties;
        if (!properties) return;
        
        let kv = new Object();
        
        for (let prop of properties) {
          if (!t.isIdentifier(prop.key)) return;
          let key = prop.key.name;
          if (!t.isLiteral(prop.value)) return;
          let value = prop.value.value;
          kv[key] = value;
        }
        
        for (let refPath of binding.referencePaths) {
          if (!refPath.parentPath) return;
          let parentNode = refPath.parentPath.node;
          if (!t.isMemberExpression(parentNode)) return;
          let key = parentNode.property.name;
          let value = kv[key];
          refPath.parentPath.replaceWith(t.valueToNode(value));
        }
        
        path.remove();
      } 
    }
  };
}

We target VariableDeclarator nodes with a visitor function, do some quick checks that variable declarator we have found is declaring a constant object. Then we gather key-value pairs into a new object while checking that each value is a literal (e.g. some numeric constant). Next we iterate across references by using Babel scope and binding (see previous post on this) to replace an object reference (MemberExpression node) with the final value (we get it by saying refPath.parentPath.node as each entry in Binding.referencePaths point to Identifier node that does the referencing). Finally, we remove the no longer needed object by removing the VariableDeclarator node from AST.

The IIFE at the bottom now looks like this:

(async () => {
  const _0x15ca68 = _0x4ad0;

  try {
    const _0x23768c = await fetch('https://api.nike.com/cic/browse/v2?queryid=products&anonymousId=A7CA2A92E0F04E570767C7810552BBC5&country=au&endpoint=%2Fproduct_feed%2Frollup_threads%2Fv2%3Ffilter%3Dmarketplace(AU)%26filter%3Dlanguage(en-GB)%26filter%3DemployeePrice(true)%26filter%3DattributeIds(8529ff38-7de8-4f69-973c-9fdbfb102ed2%2C16633190-45e5-4830-a068-232ac7aea82c)%26anchor%3D24%26consumerChannelId%3Dd9a5bc42-4b9c-4976-858a-f159cf99c647%26count%3D24&language=en-GB&localizedRangeStr=%7BlowestPrice%7D%E2%80%94%7BhighestPrice%7D', {
      'headers': {
        'authority': _0x15ca68(0),
        'accept': _0x15ca68(1),
        'accept-language': _0x15ca68(2),
        'cache-control': 'no-cache',
        'origin': _0x15ca68(3),
        'pragma': 'no-cache',
        'referer': _0x15ca68(4),
        'sec-ch-ua': '\x22Not_A\x20Brand\x22;v=\x2299\x22,\x20\x22Google\x20Chrome\x22;v=\x22109\x22,\x20\x22Chromium\x22;v=\x22109\x22',
        'sec-ch-ua-mobile': '?0',
        'sec-ch-ua-platform': _0x15ca68(5),
        'sec-fetch-dest': _0x15ca68(6),
        'sec-fetch-mode': _0x15ca68(7),
        'sec-fetch-site': 'same-site',
        'user-agent': 'Mozilla/5.0\x20(Macintosh;\x20Intel\x20Mac\x20OS\x20X\x2010_15_7)\x20AppleWebKit/537.36\x20(KHTML,\x20like\x20Gecko)\x20Chrome/109.0.0.0\x20Safari/537.36'
      }
    });

    const _0x1499a9 = await _0x23768c[_0x15ca68(8)]();

    console['log'](_0x1499a9);
  } catch (_0x307ab8) {
    console[_0x15ca68(9)](_0x307ab8);
  }
})();

Screenshot 8

Well, that’s a bit better, but we still got some indirection in the code. There are some variables/constants that do nothing but just serve as another name for something else in the code. One culprit is _0x15ca68 in the IIFE, but there’s some more in the functions further up. What we can do is get rid of these secondary variables and replace all references to them with references to the original value/object/function it points to.

The AST transform to do this is as follow:

export default function (babel) {
  const { types: t } = babel;

  return {
    name: "ast-transform", // not required
    visitor: {
      VariableDeclarator(path) {
        let node = path.node;
        if (!node.init) return;
        if (!t.isIdentifier(node.init)) return;
        let scope = path.scope;
        let binding = scope.getBinding(node.id.name);
        
        for (let refPath of binding.referencePaths) {
          refPath.replaceWith(node.init);
        }
        path.remove();
      } 
    }
  };
}

To recognise a VariableDeclarator node that introduces a secondary identifier, we merely have to check if there’s a Identifier node at init property. If that is the case, we replace all references to this node with Identifier at init property, so that all the referencing code uses the name of primary variable/constant/function. Then we remove the redundant variable declaration from the code.

Now the code is further simplified and looks like this:

function _0x5d79() {
  const _0x3d5a8f = ['yxbPlM5PA2uUy29T', 'kI8Q', 'zw4Tr0iSzw4Tvvm7Ct0WlJKSzw47Ct0WlJG', 'Ahr0Chm6lY93D3CUBMLRzs5JB20', 'Ahr0Chm6lY93D3CUBMLRzs5JB20V', 'iM1Hy09tiG', 'zw1WDhK', 'y29YCW', 'ANnVBG', 'Bg9N'];

  _0x5d79 = function () {
    return _0x3d5a8f;
  };

  return _0x5d79();
}

function _0x4ad0(_0x5d79db, _0x4ad0a1) {
  const _0x150e0f = _0x5d79();

  _0x4ad0 = function (_0x2ee979, _0x4a74f2) {
    _0x2ee979 = _0x2ee979 - 0x0;
    let _0x299466 = _0x150e0f[_0x2ee979];

    if (_0x4ad0['sjCySo'] === undefined) {
      var _0x1cb49b = function (_0x307ab8) {
        const _0x4d32cc = 'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789+/=';
        let _0x19f00a = '';
        let _0xf75b57 = '';

        for (let _0x1806d0 = 0x0, _0x425bd6, _0x1bc5e5, _0x152116 = 0x0; _0x1bc5e5 = _0x307ab8['charAt'](_0x152116++); ~_0x1bc5e5 && (_0x425bd6 = _0x1806d0 % 0x4 ? _0x425bd6 * 0x40 + _0x1bc5e5 : _0x1bc5e5, _0x1806d0++ % 0x4) ? _0x19f00a += String['fromCharCode'](0xff & _0x425bd6 >> (-0x2 * _0x1806d0 & 0x6)) : 0x0) {
          _0x1bc5e5 = _0x4d32cc['indexOf'](_0x1bc5e5);
        }

        for (let _0x4858d8 = 0x0, _0x24c07d = _0x19f00a['length']; _0x4858d8 < _0x24c07d; _0x4858d8++) {
          _0xf75b57 += '%' + ('00' + _0x19f00a['charCodeAt'](_0x4858d8)['toString'](0x10))['slice'](-0x2);
        }

        return decodeURIComponent(_0xf75b57);
      };

      _0x4ad0['rQzPwo'] = _0x1cb49b;
      _0x5d79db = arguments;
      _0x4ad0['sjCySo'] = !![];
    }

    const _0x29a87a = _0x150e0f[0x0];

    const _0x23768c = _0x2ee979 + _0x29a87a;

    const _0x1499a9 = _0x5d79db[_0x23768c];

    if (!_0x1499a9) {
      _0x299466 = _0x4ad0['rQzPwo'](_0x299466);
      _0x5d79db[_0x23768c] = _0x299466;
    } else {
      _0x299466 = _0x1499a9;
    }

    return _0x299466;
  };

  return _0x4ad0(_0x5d79db, _0x4ad0a1);
}

(async () => {
  try {
    const _0x23768c = await fetch('https://api.nike.com/cic/browse/v2?queryid=products&anonymousId=A7CA2A92E0F04E570767C7810552BBC5&country=au&endpoint=%2Fproduct_feed%2Frollup_threads%2Fv2%3Ffilter%3Dmarketplace(AU)%26filter%3Dlanguage(en-GB)%26filter%3DemployeePrice(true)%26filter%3DattributeIds(8529ff38-7de8-4f69-973c-9fdbfb102ed2%2C16633190-45e5-4830-a068-232ac7aea82c)%26anchor%3D24%26consumerChannelId%3Dd9a5bc42-4b9c-4976-858a-f159cf99c647%26count%3D24&language=en-GB&localizedRangeStr=%7BlowestPrice%7D%E2%80%94%7BhighestPrice%7D', {
      'headers': {
        'authority': _0x4ad0(0),
        'accept': _0x4ad0(1),
        'accept-language': _0x4ad0(2),
        'cache-control': 'no-cache',
        'origin': _0x4ad0(3),
        'pragma': 'no-cache',
        'referer': _0x4ad0(4),
        'sec-ch-ua': '\x22Not_A\x20Brand\x22;v=\x2299\x22,\x20\x22Google\x20Chrome\x22;v=\x22109\x22,\x20\x22Chromium\x22;v=\x22109\x22',
        'sec-ch-ua-mobile': '?0',
        'sec-ch-ua-platform': _0x4ad0(5),
        'sec-fetch-dest': _0x4ad0(6),
        'sec-fetch-mode': _0x4ad0(7),
        'sec-fetch-site': 'same-site',
        'user-agent': 'Mozilla/5.0\x20(Macintosh;\x20Intel\x20Mac\x20OS\x20X\x2010_15_7)\x20AppleWebKit/537.36\x20(KHTML,\x20like\x20Gecko)\x20Chrome/109.0.0.0\x20Safari/537.36'
      }
    });

    const _0x1499a9 = await _0x23768c[_0x4ad0(8)]();

    console['log'](_0x1499a9);
  } catch (_0x307ab8) {
    console[_0x4ad0(9)](_0x307ab8);
  }
})();

Screenshot 9

In the IIFE we got direct calls to decoder function, as well as slightly simplified code in the decoder function itself. We’re getting close now. Let us rename some of the identifiers to more readable form by applying another simple transformation:

const betterNames = {
  "_0x4ad0": "decode",
  "_0x3d5a8f": "encodedStrings",
  "_0x5d79": "getEncodedStrings",
  "_0x4d32cc": "base64Alphabet",
  "_0x23768c": "response",
  "_0x1499a9": "json"
};

export default function (babel) {
  const { types: t } = babel;

  return {
    name: "ast-transform", // not required
    visitor: {
      Identifier(path) {
        let name = path.node.name;
        if (name in betterNames) {
          let newName = betterNames[name];
          let newIdentifier = t.identifier(newName);
          
          let scope = path.scope;
          let binding = scope.getBinding(name);
          
          for (let refPath of binding.referencePaths) {
            refPath.replaceWith(newIdentifier);
          }
          
          path.replaceWith(newIdentifier);
        }
      }
    }
  };
}

The code now looks like this:

function getEncodedStrings() {
  const encodedStrings = ['yxbPlM5PA2uUy29T', 'kI8Q', 'zw4Tr0iSzw4Tvvm7Ct0WlJKSzw47Ct0WlJG', 'Ahr0Chm6lY93D3CUBMLRzs5JB20', 'Ahr0Chm6lY93D3CUBMLRzs5JB20V', 'iM1Hy09tiG', 'zw1WDhK', 'y29YCW', 'ANnVBG', 'Bg9N'];

  getEncodedStrings = function () {
    return encodedStrings;
  };

  return getEncodedStrings();
}

function decode(_0x5d79db, _0x4ad0a1) {
  const _0x150e0f = getEncodedStrings();

  decode = function (_0x2ee979, _0x4a74f2) {
    _0x2ee979 = _0x2ee979 - 0x0;
    let _0x299466 = _0x150e0f[_0x2ee979];

    if (decode['sjCySo'] === undefined) {
      var _0x1cb49b = function (_0x307ab8) {
        const base64Alphabet = 'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789+/=';
        let _0x19f00a = '';
        let _0xf75b57 = '';

        for (let _0x1806d0 = 0x0, _0x425bd6, _0x1bc5e5, _0x152116 = 0x0; _0x1bc5e5 = _0x307ab8['charAt'](_0x152116++); ~_0x1bc5e5 && (_0x425bd6 = _0x1806d0 % 0x4 ? _0x425bd6 * 0x40 + _0x1bc5e5 : _0x1bc5e5, _0x1806d0++ % 0x4) ? _0x19f00a += String['fromCharCode'](0xff & _0x425bd6 >> (-0x2 * _0x1806d0 & 0x6)) : 0x0) {
          _0x1bc5e5 = base64Alphabet['indexOf'](_0x1bc5e5);
        }

        for (let _0x4858d8 = 0x0, _0x24c07d = _0x19f00a['length']; _0x4858d8 < _0x24c07d; _0x4858d8++) {
          _0xf75b57 += '%' + ('00' + _0x19f00a['charCodeAt'](_0x4858d8)['toString'](0x10))['slice'](-0x2);
        }

        return decodeURIComponent(_0xf75b57);
      };

      decode['rQzPwo'] = _0x1cb49b;
      _0x5d79db = arguments;
      decode['sjCySo'] = !![];
    }

    const _0x29a87a = _0x150e0f[0x0];
    const response = _0x2ee979 + _0x29a87a;
    const json = _0x5d79db[response];

    if (!json) {
      _0x299466 = decode['rQzPwo'](_0x299466);
      _0x5d79db[response] = _0x299466;
    } else {
      _0x299466 = json;
    }

    return _0x299466;
  };

  return decode(_0x5d79db, _0x4ad0a1);
}

(async () => {
  try {
    const response = await fetch('https://api.nike.com/cic/browse/v2?queryid=products&anonymousId=A7CA2A92E0F04E570767C7810552BBC5&country=au&endpoint=%2Fproduct_feed%2Frollup_threads%2Fv2%3Ffilter%3Dmarketplace(AU)%26filter%3Dlanguage(en-GB)%26filter%3DemployeePrice(true)%26filter%3DattributeIds(8529ff38-7de8-4f69-973c-9fdbfb102ed2%2C16633190-45e5-4830-a068-232ac7aea82c)%26anchor%3D24%26consumerChannelId%3Dd9a5bc42-4b9c-4976-858a-f159cf99c647%26count%3D24&language=en-GB&localizedRangeStr=%7BlowestPrice%7D%E2%80%94%7BhighestPrice%7D', {
      'headers': {
        'authority': decode(0),
        'accept': decode(1),
        'accept-language': decode(2),
        'cache-control': 'no-cache',
        'origin': decode(3),
        'pragma': 'no-cache',
        'referer': decode(4),
        'sec-ch-ua': '\x22Not_A\x20Brand\x22;v=\x2299\x22,\x20\x22Google\x20Chrome\x22;v=\x22109\x22,\x20\x22Chromium\x22;v=\x22109\x22',
        'sec-ch-ua-mobile': '?0',
        'sec-ch-ua-platform': decode(5),
        'sec-fetch-dest': decode(6),
        'sec-fetch-mode': decode(7),
        'sec-fetch-site': 'same-site',
        'user-agent': 'Mozilla/5.0\x20(Macintosh;\x20Intel\x20Mac\x20OS\x20X\x2010_15_7)\x20AppleWebKit/537.36\x20(KHTML,\x20like\x20Gecko)\x20Chrome/109.0.0.0\x20Safari/537.36'
      }
    });
    const json = await response[decode(8)]();
    console['log'](json);
  } catch (_0x307ab8) {
    console[decode(9)](_0x307ab8);
  }
})();

Screenshot 10

One minor issue is that it incorrectly renamed something to response and json in the decode() function, but we can afford not to worry about this, as this function is going away soon.

We are getting close to our grand finale, but there’s one small thing we would like to fix. In the decode() function there is a single call to getEncodedStrings() that returns a hardcoded array of concealed strings. The getEncodedStrings() function is not called anywhere else, so we can do one more quick transform to replace a call to this function (CallExpression object) with the array itself (ArrayExpression object). We write another quick transform to do so and also remove the getEncodedStrings() function:

export default function (babel) {
  const { types: t } = babel;

  return {
    name: "ast-transform", // not required
    visitor: {
      FunctionDeclaration(path) {
        let node = path.node;
        if (node.id.name === "getEncodedStrings") {
          let encodedStringArrayExpr = node.body.body[0].declarations[0].init;
          let scope = path.scope;
          let binding = scope.getBinding(node.id.name);
          
          for (let refPath of binding.referencePaths) {
            if (refPath.parentPath.node.type === "CallExpression") {
              refPath.parentPath.replaceWith(encodedStringArrayExpr);
            }
          }
          
          path.remove();
        }
      }        
    }
  };
}

The function that returns an array is gone and array is now hardcoded into decode() function:

function decode(_0x5d79db, _0x4ad0a1) {
  const _0x150e0f = ['yxbPlM5PA2uUy29T', 'kI8Q', 'zw4Tr0iSzw4Tvvm7Ct0WlJKSzw47Ct0WlJG', 'Ahr0Chm6lY93D3CUBMLRzs5JB20', 'Ahr0Chm6lY93D3CUBMLRzs5JB20V', 'iM1Hy09tiG', 'zw1WDhK', 'y29YCW', 'ANnVBG', 'Bg9N'];

Screenshot 11

What we have achieved is the following:

String decoding logic is now contained into a single function - decode(). The code of this function could be regenerated with Babel and passed into Function.call() or Node’s vm API to get decoded values.
We have explicit argument values to use with decode() in the IIFE.

We are now prepared to undo the string concealing. For the sake of simplicity we will refrain from regenerating code for decode() function and will copy-paste it into AST Explorer’s transform pane instead. The final transform is as follows:

function decode(_0x5d79db, _0x4ad0a1) {
  ...
}

export default function (babel) {
  const { types: t } = babel;

  return {
    name: "ast-transform", // not required
    visitor: {
      CallExpression(path) {
        let node = path.node;
        if (node.callee.name != "decode") return;
        if (!node.arguments) return;
        if (node.arguments.length != 1) return;
        let arg = node.arguments[0].value;
        let decodedStr = decode(arg);
        path.replaceWith(t.valueToNode(decodedStr));
      },
      FunctionDeclaration(path) {
        if (path.node.id.name === "decode") path.remove();
      }
    }
  };
}

This yields a deobfuscated version of code that is very similar to what we had in the initial snippet:

(async () => {
  try {
    const response = await fetch('https://api.nike.com/cic/browse/v2?queryid=products&anonymousId=A7CA2A92E0F04E570767C7810552BBC5&country=au&endpoint=%2Fproduct_feed%2Frollup_threads%2Fv2%3Ffilter%3Dmarketplace(AU)%26filter%3Dlanguage(en-GB)%26filter%3DemployeePrice(true)%26filter%3DattributeIds(8529ff38-7de8-4f69-973c-9fdbfb102ed2%2C16633190-45e5-4830-a068-232ac7aea82c)%26anchor%3D24%26consumerChannelId%3Dd9a5bc42-4b9c-4976-858a-f159cf99c647%26count%3D24&language=en-GB&localizedRangeStr=%7BlowestPrice%7D%E2%80%94%7BhighestPrice%7D', {
      'headers': {
        'authority': "api.nike.com",
        'accept': "*/*",
        'accept-language': "en-GB,en-US;q=0.9,en;q=0.8",
        'cache-control': 'no-cache',
        'origin': "https://www.nike.com",
        'pragma': 'no-cache',
        'referer': "https://www.nike.com/",
        'sec-ch-ua': '\x22Not_A\x20Brand\x22;v=\x2299\x22,\x20\x22Google\x20Chrome\x22;v=\x22109\x22,\x20\x22Chromium\x22;v=\x22109\x22',
        'sec-ch-ua-mobile': '?0',
        'sec-ch-ua-platform': "\"macOS\"",
        'sec-fetch-dest': "empty",
        'sec-fetch-mode': "cors",
        'sec-fetch-site': 'same-site',
        'user-agent': 'Mozilla/5.0\x20(Macintosh;\x20Intel\x20Mac\x20OS\x20X\x2010_15_7)\x20AppleWebKit/537.36\x20(KHTML,\x20like\x20Gecko)\x20Chrome/109.0.0.0\x20Safari/537.36'
      }
    });
    const json = await response["json"]();
    console['log'](json);
  } catch (_0x307ab8) {
    console["log"](_0x307ab8);
  }
})();

Screenshot 12

We ended up not doing comprehensive reverse engineering to discover how exactly the decode() function works. Instead, we were able to side-step that question by reusing a simplified version of string decoding code in our AST transform, thus achieving the objective of undoing the string concealing. Reversing the string decoding part could perhaps be a worthwhile exercise in itself, but in this case it would be a needless distraction.

Unifying all the AST transforms we did into a single script that Node.JS could run is left as an exercise to the reader. Another exercise could be regenerating the decode() function from AST form, so that it would not need to be copy-pasted into deobfuscator.

JavaScript AST manipulation with Babel: constant folding and propagation

Tue, 07 Feb 2023 00:00:00 +0000

Introducing constant recomputation is a commonly used code obfuscation technique. Let’s consider the following JavaScript snippet:

const fourtyTwo = 42;
const msg = "The answer is:";
console.log(msg, fourtyTwo);

We have one numeric constant (fourtyTwo) and one string constant (msg) that are passed into console.log(). Let us apply constant obfuscation by using obfuscator.io with “Numbers To Expressions” and “Split Strings” checkboxes being on.

Screenshot 1 Screenshot 2

This yields the following obfuscated version of the above snippet that we put into AST Explorer:

const fourtyTwo = 0x25fb + 0x3eb * -0x9 + -0x28e;
const msg = 'The\x20answer' + '\x20is:';
console['log'](msg, fourtyTwo);

Screenshot 3

Instead of simple integer constant and string we got some expressions that can be evaluated back into the original values, thus retaining the original behaviour of the code.

In the Babel AST we have the following. Each variable assignment is represented by VariableDeclarator object with stuff beyond the = sign at init property. The init property points to a multi-level tree made up of mostly BinaryExpression objects that represent a single arithmetic step (e.g. adding two numbers or strings together). Number negation is represented by UnaryExpression nodes. Leaves of this subtree are NumericLiteral and StringLiteral nodes.

Screenshot 4 Screenshot 5 Screenshot 6

Turning numbers into hexadecimal form is also done as a minor obfuscation that we can easily reverse. The trick is to delete .extra.raw in the NumericLiteral objects affected by this obfuscation. To accomplish this, we write a quick AST transform in the AST Explorer tool:

export default function (babel) {
  const { types: t } = babel;

  return {
    name: "ast-transform", // not required
    visitor: {
      NumericLiteral(path) {
        if (path.node.extra.raw.startsWith("0x")) delete path.node.extra.raw;
      },
    }
  };
}

This cleans up the code a bit:

const fourtyTwo = 9723 + 1003 * -9 + -654;
const msg = 'The\x20answer' + '\x20is:';
console['log'](msg, fourtyTwo);

Screenshot 7

After this sidenote, we now want to use Babel to turn these expressions back into original values. This can be done by doing a code transformation known as constant folding. It is commonly performed by compilers during code optimisation step. Since Babel is a bit like compiler, it can perform it for us without the need of having to implement comprehensive JS expression handling algorithms or calling eval(). The key is to use NodePath.evaluate() method in the visitor function. It returns a JavaScript result object with confident flag to let us know if it was able to evaluate the expression in a way that the result is conclusive (that is not always the case, as expressions can be more complicated than what we have here and involve variables that may change during runtime).

Once again, we can prototype our deobfuscation trick in AST Explorer. A simple example of constant folding would be like this:

export default function (babel) {
  const { types: t } = babel;

  return {
    name: "ast-transform", // not required
    visitor: {
      BinaryExpression(path) {
        let evaluated = path.evaluate();
        if (!evaluated) return;
        if (!evaluated.confident) return;
        
        let value = evaluated.value;
        let valueNode = t.valueToNode(value);
        if (!t.isLiteral(valueNode)) return;
        
        path.replaceWith(valueNode);
      }
    }
  };
}

We target BinaryExpression nodes with our visitor function and just call the evaluate() method on associated NodePath object. If the result passes some simple sanity checks (the aforementioned confident flag and ability to convert the resulting value into a Literal) we replace the BinaryExpression node with newly generated NumericLiteral/StringLiteral node containing the same value that JS interpreter would have to compute.

The resulting deobfuscated code is as follows:

const fourtyTwo = 42;
const msg = "The answer is:";
console['log'](msg, fourtyTwo);

Screenshot 8

So far, so good. But what if we wanted to take this a bit further? There’s really no need to have two named constants that are only used in console.log() call, when we can just call it with the actual values. To replace references to msg and fourtyTwo in the last statement we have to apply another code optimisation technique - constant propagation. Let me show a quick AST transform to demonstrate it.

Let’s put the following snippet into AST Explorer:

const fourtyTwo = 42;
const msg = "The answer is:";
console.log(msg, fourtyTwo);

let t;
t = Math.floor(Date.now() / 1000);
console.log(t);

Since t here is not a constant it should be left unmodified.

Since code obfuscation solutions may introduce scope confusion by creating multiple identifiers with same name in multiple lexical scopes, it’s insufficient to substitute variable names into values based on name alone. We can use Babel scopes and bindings to perform substitution in a smarter way. The transform would be like this:

export default function (babel) {
  const { types: t } = babel;

  return {
    name: "ast-transform", // not required
    visitor: {
      VariableDeclarator(path) {
        if (path.node.init == null) return;
        if (!t.isLiteral(path.node.init)) return;
        const binding = path.scope.getBinding(path.node.id.name);
        if (!binding.constant) return;
        for (let i = 0; i < binding.referencePaths.length; i++) {
          let refPath = binding.referencePaths[i];
          refPath.replaceWith(path.node.init);
        }
        path.remove();
      } 
    }
  };
}

We target the VariableDeclarator nodes. In the visitor function we perform some applicability checks. First we check if it declares a variable with literal value by checking the init field. Next, we get a scope binding object and check the constant Boolean property. If these checks pass, we replace all the references to this variable with the literal node containing the value. Lastly, we remove the constant declaration as we no longer need it in the code. For more on scopes and bindings, see a previous post.

The modified snippet is now the following:

console.log("The answer is:", 42);
let t;
t = Math.floor(Date.now() / 1000);
console.log(t);

Screenshot 9

One last thing… In the above examples, a const keyword was used when declaring a constant, but it does not really matter for the purposes of this kind of (de)obfuscation. All that matters is that a value is established once and does not change later in the code.

JavaScript AST manipulation with Babel: AST modification APIs

Mon, 30 Jan 2023 00:00:00 +0000

In previous posts about using Babel for JavaScript deobfuscation, we have used NodePath.replaceWith() method to replace one node with another and NodePath.remove() to remove a single node. Since AST and it’s elements are mutable, we can also modify the AST without traversing it. But there is more to learn about AST modification than what we have seen before. We will go through some more AST modification APIs that should prove to be useful when developing JS deobfuscators.

NodePath.replaceWithMultiple() replaces a single node with a list of nodes.

NodePath.replaceWithSourceString() is a convenience method that takes a source string and parses it for you when replacing the node. However this is inefficient and not recommended. There are better ways to convert a source string (or template) into AST subtree.

We can also insert a node before or after the current node. NodePath.insertBefore() and NodePath.insertAfter() methods can be used for that.

Sometimes there are AST node property arrays that contain multiple child nodes. You may want to to add a single new entry to beginning or end of that array. NodePath.unshiftContainer() prepends a new node and NodePath.pushContainer() appends it to the end of array. However it might be bit confusing on how to use these methods. Consider the following snippet that we want to change:

function say_abc() {
  console.log("a");
  console.log("b");
  console.log("c");
}

say_abc();

Let’s suppose we want to add debugger; before and after the console.log() calls in the function. When we parse this code into AST, we can see that we have a following subtree:

FunctionDeclaration
body: BlockStatement
- body: [] (ExpressionStatements)

Screenshot 1

We want to modify the function body, so we will get a NodePath for the BlockStatement at body property of FunctionDeclaration. The exact array we want to modify is body of BlockStatement. Thus we call the above methods on BlockStatement object with two arguments: 1) node property name (body) and 2) the node object we want to insert. In AST Explorer we develop the following transform:

export default function (babel) {
  const { types: t } = babel;

  return {
    name: "ast-transform", // not required
    visitor: {
      FunctionDeclaration(path) {
        let body = path.get('body');
        body.unshiftContainer('body', t.debuggerStatement());
        body.pushContainer('body', t.debuggerStatement());
      }
    }
  };
}

As intended, this modifies the toy code snippet into the following:

function say_abc() {
  debugger;
  console.log("a");
  console.log("b");
  console.log("c");
  debugger;
}

say_abc();

Screenshot 2 Screenshot 3

If for some reason you want to skip traversal of children of current node (e.g. to avoid infinite loop when nodes are being added) you can call NodePath.skip(). To completely stop the AST traversal, you can call NodePath.stop().

JavaScript AST manipulation with Babel: untangling scope confusion

Wed, 25 Jan 2023 00:00:00 +0000

In computer programming, a lexical scope of an identifier (name of function, class, variable, constant, etc.) is area within the code where that identifier can be used. Some identifiers have global scope meaning they can be used in the entire program. Others have narrowed-down scope that is limited to one single function or code block between curly braces. Identifier names can be reused between scopes - conflict is resolved by preferring the innermost scope for a given identifier. Lexical scopes are set in stone as the program is written.

JavaScript also has some support for dynamic scoping that is resolved during program runtime, but let us put that aside for time being.

Introducing redundant lexical scopes with repeating identifiers is often done for code obfuscation purposes, as it makes code hard to read and understand. This motivates us to learn how to gain clarity when such scope obfuscation technique is applied on client-side JavaScript code that we want to reverse engineer.

Since Babel parses the JS code into data model representing logical structure of the code, it also has a capability to represent lexical scopes via Scope object. Furthermore, Babel is able to handle scope bindings - relations between identifiers and scopes. Through a scope binding we can find the usage of variable name or some other identifier to track down where in the code that identifier is being referenced. Thus we have all the tools to programmatically disambiguate between multiple identifiers across nested scopes and convert the code into a more readable form.

Let us use ASTExplorer with Chrome DevTools to explore scopes and bindings. Put the following JS snippet in there:

let x = 10;
let y = 20;

{
  let y = 30;
  console.log(x + y);
}

console.log(x + y);

Screenshot 1

We can see that there is no data about scopes and bindings in the final AST that Babel parser outputs. There’s a small trick to getting to that in ASTExplorer. First, we need to turn on the tranform sections and make a small AST visitor function (within provided boilerplate) that prints scopes/bindings for the AST nodes we are interested in. Let us see the Scope object for the root AST node - Program:

export default function (babel) {
  const { types: t } = babel;

  return {
    name: "ast-transform", // not required
    visitor: {
      Program(path) {
        console.log(path.scope);
      }
    }
  };
}

To see what console.log() prints, we must switch on Chrome DevTools panel and select Console tab. Some properties or interest are:

block - AST node for code block (BlockStatement) or something else (e.g. Program) that limits the scope.
bindings - mapping of identifier names to Binding objects associated with the current scope.
globals - list of AST nodes that are associated with the global scope; current scope inherits them.
path - a NodePath object that references back to the visited node and its position within AST.

Screenshot 2

Some interesting properties of Binding object are:

identifier - an Identifier node for the identifier in question (constant name in this case)
path - a NodePath object for AST node that introduces this identifier into scope.
referenced - flag to check if identifier has been referenced.
references - number of references to identifier within the scope.
referencePaths - an array of NodePath objects that are associated with the current identifier being referenced within the scope.

Screenshot 3

Note that for x we have 2 references and for y we have only one reference. That should be expected, as in the inner lexical scope there is new declaration of y that overrides the declaration of y in the outer scope.

These two constants that are both named y require further attention. Let us take a deeper look into their scopes and bindings. Update the visitor function in the bottom left corner to the following:

export default function (babel) {
  const { types: t } = babel;

  return {
    name: "ast-transform", // not required
    visitor: {
      VariableDeclarator(path) {
        if (path.node.id.name == "y") {
          console.log(path.scope);
          console.log("PARENT", path.scope.parent);
        }
      }
    }
  };
}

We are visiting VariableDeclarator nodes, but we are only interested in the ones that declare y. For these, we log the current scope and parent scope. In our example, parent scope is the scope for entire program and child scope is within the curly braces.

Screenshot 4

To check for identifier name duplication, we can look into parent scope bindings and see if there’s a binding for another identitifier with the same name that is being overridden. To prevent identifiers from being overriden we can rename the new identifiers in child scopes. Let me show an example of Babel transform that would do this:

export default function (babel) {
  const { types: t } = babel;

  return {
    name: "ast-transform", // not required
    visitor: {
      VariableDeclarator(path) {
        let idName = path.node.id.name;
        let parentScope = path.scope.parent;
        if (!parentScope) return;
        if (parentScope.getBinding(idName)) {
          let newName = path.scope.generateUidIdentifier(idName);
          path.scope.rename(idName, newName.name);
        }
      }
    }
  };
}

This transform modifies the snippet so that constant name duplication goes away:

let x = 10;
let y = 20;
{
  let _y = 30;
  console.log(x + _y);
}
console.log(x + y);

We used Scope.rename() method to rename the constant within the scope (covering both the declaration and statement that uses it) to a name that does not duplicate with constant name in a parent scope. To save us from the hassle of generating unique names, Babel provides us Scope.generateUidIdentifier() method that we used to get a new Identifier node. Fixing name duplication turned out to be just a few lines of code.

Note that in some cases you may need to call crawl() on current or parent scope to make it update all the JS statements that reference the identifier that has just been renamed.

Let us get more serious now by applying DefendJS scope obfuscation feature:

$ cp 1.js 6.js  
$ defendjs --input 6.js --features scope --output .

The obfuscated version of the code is now like this:

(function () {
    {
        {
            function b(a, b) {
                return Array.prototype.slice.call(a).concat(Array.prototype.slice.call(b));
            }
            function c() {
                var a = arguments[0], c = Array.prototype.slice.call(arguments, 1);
                var b = function () {
                    return a.apply(this, c.concat(Array.prototype.slice.call(arguments)));
                };
                b.prototype = a.prototype;
                return b;
            }
            function d(a, b) {
                return Array.prototype.slice.call(a, b);
            }
            function e(b) {
                var c = {};
                for (var a = 0; a < b.length; a += 2) {
                    c[b[a]] = b[a + 1];
                }
                return c;
            }
            function f(a) {
                return a.map(function (a) {
                    return String.fromCharCode(a & ~0 >>> 16) + String.fromCharCode(a >> 16);
                }).join('');
            }
            function g() {
                return String.fromCharCode.apply(null, arguments);
            }
        }
        var a = [];
        a[0] = 10;
        a[1] = 20;
        a[2] = 30;
        console.log(a[0] + a[2]);
        console.log(a[0] + a[1]);
    }
}())

Applying the current Babel transform does rename some variables in the junk functions, but code stays mostly the same:

(function () {
  {
    {
      function b(a, b) {
        return Array.prototype.slice.call(a).concat(Array.prototype.slice.call(b));
      }

      function c() {
        var _a = arguments[0],
            _c = Array.prototype.slice.call(arguments, 1);

        var _b = function () {
          return _a.apply(this, _c.concat(Array.prototype.slice.call(arguments)));
        };

        _b.prototype = _a.prototype;
        return _b;
      }

      function d(a, b) {
        return Array.prototype.slice.call(a, b);
      }

      function e(b) {
        var _c2 = {};

        for (var _a2 = 0; _a2 < b.length; _a2 += 2) {
          _c2[b[_a2]] = b[_a2 + 1];
        }

        return _c2;
      }

      function f(a) {
        return a.map(function (a) {
          return String.fromCharCode(a & ~0 >>> 16) + String.fromCharCode(a >> 16);
        }).join('');
      }

      function g() {
        return String.fromCharCode.apply(null, arguments);
      }
    }
    var _a3 = [];
    _a3[0] = 10;
    _a3[1] = 20;
    _a3[2] = 30;
    console.log(_a3[0] + _a3[2]);
    console.log(_a3[0] + _a3[1]);
  }
})();

Can we remove these junk functions that do not get called anywhere? Let us try reworking the transform.

Remember that Binding object also keeps track of references to the identifier in question? We can remove these functions based on them not being referenced anywhere:

export default function (babel) {
  const { types: t } = babel;

  return {
    name: "ast-transform", // not required
    visitor: {
      FunctionDeclaration(path) {
        let fnName = path.node.id.name;
        let binding = path.scope.getBinding(fnName);
        if (!binding.referenced) path.remove();
      }
    }
  };
}

Now the code in the output is like this:

(function () {
  {
    {
      function b(a, b) {
        return Array.prototype.slice.call(a).concat(Array.prototype.slice.call(b));
      }

      function c() {
        var a = arguments[0],
            c = Array.prototype.slice.call(arguments, 1);

        var b = function () {
          return a.apply(this, c.concat(Array.prototype.slice.call(arguments)));
        };

        b.prototype = a.prototype;
        return b;
      }
    }
    var a = [];
    a[0] = 10;
    a[1] = 20;
    a[2] = 30;
    console.log(a[0] + a[2]);
    console.log(a[0] + a[1]);
  }
})();

All but two junk functions got removed. A closer look into the remaining functions reveals that they are referencing themselves. Furthermore, we ran into a weird feature of JavaScript: var statements that are able to transcend the child scope and set the variable in a more global scope. For example, the following code snippet from Mozilla developer portal counter-intuitively prints 2 twice:

var x = 1;

if (x === 1) {
  var x = 2;

  console.log(x);
  // Expected output: 2
}

console.log(x);
// Expected output: 2

In the currently deobfuscated code b() references itself and c() references b() (and also itself) through var statement.

At the moment I can’t think of a solution to resolve this kind of circular references, but we already made some progress. We can see that both remaining functions are not called anywhere and don’t really do anything. Thus it is safe to disregard them for time being.

Note however that Babel does not track scopes across files. It can only work with one file at a time.

JavaScript AST manipulation with Babel: 3 ways to create nodes and subtrees

Mon, 23 Jan 2023 00:00:00 +0000

When doing Javascript deobfuscation work at AST level one will need to create new parts of AST for replacing the existing parts of AST being processed. We will go through 3 ways of doing that.

For the sake of an example, let us try to build an AST for the following JS snippet:

debugger;
console.log("!")
debugger;

Babel parses this into two DebuggerStatement nodes and and one ExpressionStatement node.

Screenshot

We don’t need to worry about File and Program objects and should only focus on the program body that contains the following child nodes:

DebuggerStatement
ExpressionStatement
- expression: CallExpression
  - callee: MemberExpression
    - object: Identifier
      - name: "console"
    - computed: false
    - property: Identifier
      - name: "log"
- arguments: []
  1. StringLiteral
    - extra:
      - rawValue: "!"
    - value: "!"
DebuggerStatement

Once we build these three statements, we can replace some part of AST with them. That will be shown as well.

Instantiate node objects, arrange them into tree

In software development there is often a tradeoff to be made between simplicity of the code and it’s flexibility. We will start with the least simple, but most flexible approach - explicitly instantiating AST node objects and arranging them in a way that recreates the AST subtree above.

We need to refer to spec for Babel AST objects and their properties in Babel codebase file packages/babel-parser/ast/spec.md. Furthermore, we need to check node creation APIs in @babel/types documentation.

By using the information in these two documents, we can write the following code:

const parser = require("@babel/parser");
const generate = require("@babel/generator").default;
const t = require("@babel/types");

const debugStmt1 = t.debuggerStatement();

const consoleId = t.identifier("console");
const logId = t.identifier("log");
const stringLiteral = t.stringLiteral("!");
const callee = t.memberExpression(consoleId, logId);
const callExpr = t.callExpression(callee, [ stringLiteral ]);
const exprStmt = t.expressionStatement(callExpr);

const debugStmt2 = { type: 'DebuggerStatement' };

const body = [ debugStmt1, exprStmt, debugStmt2 ];

const ast = parser.parse(""); // Parsing empty string creates AST with File and Program nodes.

ast.program.body = body;

console.log(ast);
console.log("==================");
console.log(generate(ast).code);

Note that we can build AST node objects either by using factory functions from @babel/types or by defining them as JS objects directly. Compare how debugStmt1 and debugStmt2 were created.

Running this Node.JS script prints the modified AST and the generated code:

$ node instantiate.js
Node {
  type: 'File',
  start: 0,
  end: 0,
  loc: SourceLocation {
    start: Position { line: 1, column: 0, index: 0 },
    end: Position { line: 1, column: 0, index: 0 },
    filename: undefined,
    identifierName: undefined
  },
  errors: [],
  program: Node {
    type: 'Program',
    start: 0,
    end: 0,
    loc: SourceLocation {
      start: [Position],
      end: [Position],
      filename: undefined,
      identifierName: undefined
    },
    sourceType: 'script',
    interpreter: null,
    body: [ [Object], [Object], [Object] ],
    directives: []
  },
  comments: []
}
==================
debugger;
console.log("!");
debugger;

Thus we have demonstrated a very basic example of JS code generation with Babel.

Use Babel template

@babel/template module lets us define a code template, parametrise it with AST nodes and create a subtree from that.

Let us generate the JS snippet using this approach:

const parser = require("@babel/parser");
const generate = require("@babel/generator").default;
const t = require("@babel/types");
const template = require("@babel/template").default;

const exampleTempl = template(`
debugger;
console.log(%%msg%%);
debugger;
`);

const body = exampleTempl({
    msg: t.stringLiteral("!")
});

const ast = parser.parse("");

ast.program.body = body;

console.log(ast);
console.log("==================");
console.log(generate(ast).code);

We’re getting the same output as from the previous script.

Parse the source code

Sometimes you may want to add some code that does not depend on any context is is fairly simple. Due to recursive structure of AST we can create a small AST by parsing a JS snippet and plug it into where we want it to be in the bigger tree. This is as simple and basic as can get and I don’t think it warrants any sample code. However we must not forget that in some cases easy and simple way is the right way.

JavaScript AST manipulation with Babel: transform prototyping and plugin development

Wed, 18 Jan 2023 00:00:00 +0000

In previous posts we have went through several JavaScript obfuscation techniques and how they could be reversed by applying Abstract Syntax Tree transformations. AST manipulation is a powerful skill that is of particular importance in certain kinds of grayhat programming projects. Earlier, we have focused on how exactly AST could be changed to undo specific kinds of obfuscations. This time, however, we will take a bit broader look into Babel to see how it facilitates development of AST manipulation code that goes beyond a single AST transformation.

In one of the posts we have shown some trivial examples of AST manipulation:

Undoing hexadecimal string encoding
Converting bracket notation to dot notation
Removing excessive semi-colons from the code

First, let me show a way to prototype AST transforms in ASTExplorer. Let us put the following code in there:

console["log"]("\x41\x42\x43"); // ABC
;;;;;console.log("123");;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

Make sure that JavaScript is chosen in the language drop-down menu and @babel/parser is selected in the parser drop-down. Press the “Transform” switch and choose babelv7. Now we have two new UI sections here.

Screenshot 1

On the bottom left, there’s a transform code editing section. By default ASTExplorer gives you the following example code that reverses the order or characters in identifier names:

export default function (babel) {
  const { types: t } = babel;

  return {
    name: "ast-transform", // not required
    visitor: {
      Identifier(path) {
        path.node.name = path.node.name.split("").reverse().join("");
      }
    }
  };
}

On the bottom right section we have the JS code that is the result of this transformation.

The current transform by itself is quite useless, but we can change it to our needs. Note that there is no boilerplate code for reading the JS source code and writing it back into files. Just the AST transform that is the core of what we are doing here.

Let us change the visitor function to undo hexadecimal string encoding:

export default function (babel) {
  const { types: t } = babel;

  return {
    name: "ast-transform", // not required
    visitor: {
      StringLiteral: function (path) {
        path.node.extra.raw = '"' + path.node.extra.rawValue + '"';
      }
    }
  };
}

We can see that console["log"]("\x41\x42\x43"); from the input indeed became console["log"]("ABC"); in the output.

Screenshot 2

Since other two changes we want to make work on different node classes, we can extend the top visitor object to include them as well:

export default function (babel) {
  const { types: t } = babel;

  return {
    name: "ast-transform", // not required
    visitor: {
      StringLiteral: function (path) {
        path.node.extra.raw = '"' + path.node.extra.rawValue + '"';
      },
      CallExpression: function (path) {
        let prop = path.node.callee.property;

        if (t.isStringLiteral(prop)) {
          path.node.callee.property = t.Identifier(prop.value);
          path.node.callee.computed = false;
        }
      },
      EmptyStatement: function (path) {
        path.remove();
      }
    }
  };
}

This cleaned up the code into the following form:

console.log("ABC"); // ABC

console.log("123");

Screenshot 3

That’s nice, but our transform code is not particularly clean, as we are performing three different kinds of changes in one go. It does not matter much for simple example like this, but real-world deobfuscators will involve far more complexity and should be written with one of the best programming practices in mind: keeping stuff separate without letting it became a single intractable mess of spaghetti code. Thus we want a better structure in our code.

Babel is developed with plugin architecture in mind. Babel plugin is pluggable piece of code that does a single change to the AST being processed. In our example, there would be one plugin for undoing the hexadecimal string encoding, another for converting bracket notation to do notation and third one for removing EmptyStatement nodes. There’s also a concept of preset - a collection of plugins that make up some greater workflow, such as converting the code between JavaScript dialects or undoing the entire set of obfuscation techniques that antibot vendor is applying.

Let us save the input code from AST Explorer into input.js and write a single deobfuscator that benefits from Babel plugin architecture to a greater degree.

Consider the following code:

const fs = require("fs");
const path = require("path");

const babel = require("@babel/core");
const t = require("@babel/types");

const result = babel.transformFileSync(path.join(__dirname, 'input.js') , {
    plugins: [{
        name: "undo_hexadecimal_strings",
        visitor: {
            StringLiteral: function (path) {
                path.node.extra.raw = '"' + path.node.extra.rawValue + '"';
            }
        }
    }, {
        name: "bracket_to_dot",
        visitor: {
            CallExpression: function (path) {
                let prop = path.node.callee.property;

                if (t.isStringLiteral(prop)) {
                    path.node.callee.property = t.Identifier(prop.value);
                    path.node.callee.computed = false;
                }
            }
        }
    }, {
        name: "rm_needless_semicolons",
        visitor: {
            EmptyStatement: function (path) {
                path.remove();
            }
        }
    }]
});

console.log(result.code);

fs.writeFileSync(path.join(__dirname, 'output.js'), result.code, 'utf-8');

In our previous code in the previous posts, we had parsing, AST traversal, code generation as separate steps, which is good for understanding what’s going on in the code, but introduces a bit of repetitive boilerplate. Now we are using transformFileSync() function that does all three steps for us. It takes a file path to input JS file and a list of plugins. Each plugin is a named AST manipulation step that entails one or more visitor functions that are associated with certain types of AST nodes.

Asynchronous equivalent to this function would be transformFile(). There’s also transform(), transformSync() and transformAsync() that take JS code a string without reading it from the file.

Plugin code can be quite long and complex, which makes us want to keep it in one or more separate files. Babel allows that as well. Let us move the plugin code out of the main script by creating new files for each plugin.

First plugin file (plugin_unhex.js) would be as follows:

const babel = require("@babel/core");
const t = require("@babel/types");

module.exports = function (babel) {
    return {
        name: "undo_hexadecimal_strings",
        visitor: {
            StringLiteral: function (path) {
                path.node.extra.raw = '"' + path.node.extra.rawValue + '"';
            }
        }
    }
};

Next file, plugin_unbracket.js would be like this:

const babel = require("@babel/core");
const t = require("@babel/types");

module.exports = function (babel) {
    return {
        name: "bracket_to_dot",
        visitor: {
            CallExpression: function (path) {
                let prop = path.node.callee.property;

                if (t.isStringLiteral(prop)) {
                    path.node.callee.property = t.Identifier(prop.value);
                    path.node.callee.computed = false;
                }
            }
        }
    }
};

The third and last plugin file would contain the following code:

const babel = require("@babel/core");
const t = require("@babel/types");

module.exports = function (babel) {
    return {
        name: "rm_needless_semicolons",
        visitor: {
            EmptyStatement: function (path) {
                path.remove();
            }
        }
    }
};

By moving all three plugins to their own files, we can simplify the main script to this:

const fs = require("fs");
const path = require("path");

const babel = require("@babel/core");
const t = require("@babel/types");

const result = babel.transformFileSync(path.join(__dirname, 'input.js') , {
    plugins: [
        path.join(__dirname, 'plugin_unhex.js'),
        path.join(__dirname, 'plugin_unbracket.js'),
        path.join(__dirname, 'plugin_rm_empty.js')
    ]
});

console.log(result.code);

fs.writeFileSync(path.join(__dirname, 'output.js'), result.code, 'utf-8');

Now that is far cleaner and less nested than it was before.

Before we’re finished here, you may want to know one neat trick in ASTExplorer. When you call console.log() on a Babel object such as node, path, scope and open Chrome DevTools you can inspect various properties to debug your transformations.

Screenshot 4

To learn more about Babel plugin development, see:

Understanding Python generators for scraping and automation

Sun, 15 Jan 2023 00:00:00 +0000

On the front page of Scrapy framework there’s the following Python snippet:

import scrapy

class BlogSpider(scrapy.Spider):
    name = 'blogspider'
    start_urls = ['https://www.zyte.com/blog/']

    def parse(self, response):
        for title in response.css('.oxy-post-title'):
            yield {'title': title.css('::text').get()}

        for next_page in response.css('a.next'):
            yield response.follow(next_page, self.parse)

Note the usage of yield keyword in this code. This is different from simply returning some value from method or function. One way to look at this is that yield is a lazy equivalent of returning a value. All the code that precedes the yield keyword is only executed when the value is needed by the caller. But there’s more to it. To really understand what’s going on here, we need to understand Python generators. To understand Python generators, we need to understand iterators and to understand iterators we need to understand iterables.

So, what’s an iterable? Iterable is a collection or object that you can iterate upon in a loop, like this:

>>> l = [1, 2, 3]
>>> for x in l:
...     print(x)
... 
1
2
3

Common iterables in Python are not just lists, but sequence-like things such as sets, tuples, strings:

>>> s = set(l)
>>> for x in s:
...     print(x)
... 
1
2
3
>>> t = tuple(l)
>>> for x in t:
...     print(x)
... 
1
2
3
>>> for c in "abcd":
...     print(c)
... 
a
b
c
d

When Python runs a for-loop across an iterable, an iterator is implicitly created and used. Iterator is an object that is similar to database cursor. It is used to keep track of state (e.g. collection index) across iteration steps. Iterator is required to implement __iter()__ method to set the initial state and __next__() method to return the current element from iterable, then update the state to point to the next one. Iterators can also be created explicitly with an iter() function:

>>> it = iter(l)
>>> type(it)
<class 'list_iterator'>

Using them is simple - just replace an iterable reference with iterator reference in your for-loop:

>>> for x in it:
...     print(x)
... 
1
2
3

Once there’s nothing else to iterate, the __next__() method will raise a StopIteration exception to break the loop.

Sometimes you don’t want to use the iterator in for-loop, but just get a single value from it. This can be done with next() function:

>>> it = iter("abcd")
>>> print(next(it))
a
>>> print(next(it))
b

Now, what’s a generator? Generator is a special kind of iterator that does not go through some iterable, but lazily creates each value when it is needed. One way a generator can be created is using a generator expression:

>>> g = (i*i for i in range(10))
>>> type(g)
<class 'generator'>
>>> next(g)
0
>>> next(g)
1

Another way to create a generator is the one typically used in Scrapy projects - developing a generator function (or method) that uses the yield keyword to lazily return stuff to the caller:

>>> def gen_123():
...     print("About to yield 1")
...     yield 1
...     print("About to yield 2")
...     yield 2
...     print("About to yield 3")
...     yield 3
... 
>>> g = gen_123()
>>> type(g)
<class 'generator'>
>>> next(g)
About to yield 1
1
>>> next(g)
About to yield 2
2
>>> next(g)
About to yield 3
3
>>> next(g)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
StopIteration
>>> g = gen_123()
>>> for x in g:
...     print(x)
... 
About to yield 1
1
About to yield 2
2
About to yield 3
3

Generator functions/methods can be quite useful in web scraper and bot development as they enable us to avoid gathering scraped results in a collection, which is undesirable from performance perspective. By using generators, we can structure our code as pipeline, with each datum going through the processing steps as soon as it is ready. Generator functions can be stateful - all the variables in the function retain their values between the moments values are generated.

Consider the following example.

#!/usr/bin/python3

import csv
from urllib.parse import urljoin
from pprint import pprint

import requests
from lxml import html

FIELDNAMES = ["title", "url", "points", "discussion_url"]


def scrape_page(i):
    url = "https://news.ycombinator.com/"
    params = {"p": i}

    resp = requests.get(url, params=params)
    print(resp.url)

    tree = html.fromstring(resp.text)

    for athing_row in tree.xpath('//tr[@class="athing"]'):
        title = athing_row.xpath('.//span[@class="titleline"]/a/text()')[0]
        url = athing_row.xpath('.//span[@class="titleline"]/a/@href')[0]
        points = athing_row.xpath(
            './following-sibling::tr//span[@class="score"]/text()'
        )[0].replace(" points", "")
        discussion_url = athing_row.xpath(
            './following-sibling::tr//span[@class="age"]/a/@href'
        )[0]
        discussion_url = urljoin(resp.url, discussion_url)

        yield {
            "title": title,
            "url": url,
            "points": points,
            "discussion_url": discussion_url,
        }


def scrape(n_pages):
    for i in range(1, n_pages + 1):
        yield from scrape_page(i)


def main():
    out_f = open("hn_posts.csv", "w", encoding="utf-8")

    csv_writer = csv.DictWriter(out_f, fieldnames=FIELDNAMES, lineterminator="\n")
    csv_writer.writeheader()

    for row in scrape(3):
        pprint(row)
        csv_writer.writerow(row)

    out_f.close()


if __name__ == "__main__":
    main()

Here we scrape the first three pages of Hacker News. For each posted link we get an URL, title, number of points and link to discussion page. All of that is being written to CSV file - one row at a time. Using generator functions we can refrain from wasting resources putting each row into a list or set and then iterating across that collection when writing it out into a file. Furthermore, this is more desirable from error handling perspective, as we would not lose all the data if the script crashed (error handling has been omitted for the sake of brevity).

Both scrape_page() and scrape() are generator functions. scrape_page() scrapes an i-th page from Hacker News and generates item dictionaries one at a time. If we just wanted a single page worth of data, we could just iterate over it on the caller side and get each row when it was generated. However we have scrape() function in the middle that demonstrates iterator chaining with a fairly new Python feature - yield from. In this case, yield from is only used to avoid doing it like this:

for i in range(1, n_pages + 1)):
    for row in scrape_page(i):
        yield row

More generally, yield from establishes a connection between caller and sub-generator.

In Scrapy framework generators are used in conjunction with asynchronuous networking. Your Scrapy spider callbacks contain code that is lazily evaluated exactly when needed and use yield keyword to generate a request or data item both when it is ready and when the underlying machinery of Scrapy is able to process it.

Scraping Yahoo! Finance with Python

Sat, 14 Jan 2023 00:00:00 +0000

Yahoo! Finance is a prominent website featuring financial news, press releases, financial reports and quantitive data on various assets. We are going to go through some examples on how data could be scraped from this portal. We are going to scrape two kinds of data: fundamental infomation on how well various public companies are performing financially and stock price time series.

But first we need to have a list of stock tickers (symbols) for companies we want to gather information on. We could write a simple text file with one ticker per line, but it’s not difficult to apply a little technical trick here. If we go to Most Actives page, we get a table listing some well performing stocks. For the purpose of further development, we want the first 25 of them. A little poking around reveals that there’s a private API servicing the table and heat map here (load the page, press the “Heatmap View” element, see what appears in DevTools Network tab with XHR filter being enabled).

Screenshot 1

Data on the first 25 entries in the list can be retrieved with just a HTTPS GET request:

$ curl "https://query1.finance.yahoo.com/v1/finance/screener/predefined/saved?formatted=true&lang=en-US&region=US&scrIds=most_actives&start=0&count=250&enableSectorIndustryLabelFix=true&sortField=intradaymarketcap&sortType=DESC&corsDomain=finance.yahoo.com"

It’s just like that. No need to worry about headers or cookies or proxies.

Now we can use jq to parse the JSON we are receiving from this API:

$ curl -s "https://query1.finance.yahoo.com/v1/finance/screener/predefined/saved?formatted=true&lang=en-US&region=US&scrIds=most_actives&start=0&count=25&enableSectorIndustryLabelFix=true&corsDomain=finance.yahoo.com" |\
 jq -r ".finance.result[0].quotes[].symbol" > tickers.txt
$ wc -l tickers.txt
      25 tickers.txt

Just a single shell command got us the 25 tickers to play with.

Now let us explore the page on Yahoo! Finance for each stock, so that we could scrape the summary of fundamentals: trade volume, market cap, etc. One example is Alphabet Inc. page. Here we have a table with various numbers we want to scrape in the summary tab and a simple chart with price over time. We will scrape the former soon and get to the latter later.

How does the DOM structure look like in the summary table? Let’s select one of the labels in the table, right-click on it and choose Inspect. We get the Chrome DevTools panel with this element being highlighted in the Elements tab. Now we are looking for some trait that uniquely identifies each part of the page that we want to scrape. <td> element next to the one we selected has data-test attribute with value PREV_CLOSE-value. This looks promising.

Screenshot 2 Screenshot 3

Could it be that other value fields also have data-test attribute that we can use for finding them in a DOM? Indeed they do: it’s MARKET_CAP-value for market cap, BID-value for bid, ASK-value for asking price and so on.

Screenshot 4

However, there’s one last thing we should check. The DOM we see now in DevTools Elements tab is the one that may or may not be a result of client-side rendering. Since we are going to use plain requests for scraping, we want to see if the the same data-test attribute values appear in the HTML document that browser downloads before the rendering. We can verify that by using View Source feature.

Screenshot 5

Given what we have found, writing XPath queries for the fields is quite trivial:

$ python3
Python 3.10.8 (main, Oct 13 2022, 09:48:40) [Clang 14.0.0 (clang-1400.0.29.102)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> from lxml import html
>>> resp = requests.get("https://finance.yahoo.com/quote/GOOGL")
>>> resp
<Response [200]>
>>> tree = html.fromstring(resp.text)
>>> tree.xpath('//td[@data-test="EARNINGS_DATE-value"]/span/text()')
['Feb 02, 2023']
>>> tree.xpath('//td[@data-test="PREV_CLOSE-value"]/text()')
['91.13']

Getting a page URL given the stock symbol is easy: just append it to https://finance.yahoo.com/quote/.

The entire Python script that reads tickers from a text file and scrapes the summary table for each of them would be like this:

#!/usr/bin/python3

import csv
from pprint import pprint

import requests
from lxml import html

FIELDNAMES = [
    "ticker",
    "prev_close",
    "open",
    "bid",
    "ask",
    "days_range",
    "52_wk_range",
    "volume",
    "avg_volume",
    "market_cap",
    "beta",
    "pe",
    "eps",
    "earnings_date",
    "fw_dividend_and_yield",
    "ex_dividend_date",
    "1y_target_est",
    "url",
]


def scrape_summary(ticker):
    url = "https://finance.yahoo.com/quote/" + ticker

    xpath_by_field = {
        "prev_close": '//td[@data-test="PREV_CLOSE-value"]/text()',
        "open": '//td[@data-test="OPEN-value"]/text()',
        "bid": '//td[@data-test="BID-value"]/text()',
        "ask": '//td[@data-test="ASK-value"]/text()',
        "days_range": '//td[@data-test="DAYS_RANGE-value"]/text()',
        "52_wk_range": '//td[@data-test="FIFTY_TWO_WK_RANGE-value"]/text()',
        "volume": '//td[@data-test="TD_VOLUME-value"]/fin-streamer/text()',
        "avg_volume": '//td[@data-test="AVERAGE_VOLUME_3MONTH-value"]/text()',
        "market_cap": '//td[@data-test="MARKET_CAP-value"]/text()',
        "beta": '//td[@data-test="BETA_5Y-value"]/text()',
        "pe": '//td[@data-test="PE_RATIO-value"]/text()',
        "eps": '//td[@data-test="EPS_RATIO-value"]/text()',
        "earnings_date": '//td[@data-test="EARNINGS_DATE-value"]/span/text()',
        "fw_dividend_and_yield": '//td[@data-test="DIVIDEND_AND_YIELD-value"]/text()',
        "ex_dividend_date": '//td[@data-test="EX_DIVIDEND_DATE-value"]//text()',
        "1y_target_est": '//td[@data-test="ONE_YEAR_TARGET_PRICE-value"]/text()',
    }

    row = {"ticker": ticker, "url": url}

    resp = requests.get(url)
    print(resp.url)

    tree = html.fromstring(resp.text)

    for f in xpath_by_field.keys():
        try:
            row[f] = tree.xpath(xpath_by_field[f])[0]
        except Exception as e:
            print(f)
            print(e)

    return row


def main():
    in_f = open("tickers.txt", "r")
    tickers = in_f.read().strip().split("\n")
    in_f.close()

    out_f = open("summary.csv", "w", encoding="utf-8")
    csv_writer = csv.DictWriter(out_f, fieldnames=FIELDNAMES, lineterminator="\n")
    csv_writer.writeheader()

    for ticker in tickers:
        row = scrape_summary(ticker)
        if row is not None:
            pprint(row)
            csv_writer.writerow(row)

    out_f.close()


if __name__ == "__main__":
    main()

When launching this, we get some stock summaries saved into summary.csv file.

Screenshot 6

Now let us explore how that little stock price chart is getting data. Switch on the DevTools and go to Network tab. Press XHR button so that it shows only the requests that are launched by client-side JS code. Then press some of the time range switches on top of the chart and see what happens.

Screenshot 7 Screenshot 8

We see that there’s an API call that fetches the the price data during time window that the user chose. A list of valid ranges is included into API response as well.

We would like to scrape all the data that is needed to recreate a candle chart. Each candlestick in the chart represents the following price number for a day, hour, minute or some other interval:

Open Price - price of asset at the beginning of interval.
Close Price - price at the end of interval.
Low Price - lowest price during the interval.
High Price - highest price during the interval.

For more information, see this article.

This is indeed what the private Yahoo! Finance API gives us. Note that we get many intervals across the big range that was selected on the chart. Both candle interval and overall range can be chosen by setting the appropriate parameters.

Since this is fairly straightforward API scraping, our code is simple as well:

#!/usr/bin/python3

import csv
import sys

import requests

FIELDNAMES = ["symbol", "timestamp", "open", "close", "high", "low", "url"]


def main():
    if len(sys.argv) != 4:
        print("Usage:")
        print("{} <symbol> <interval> <range>".format(sys.argv[0]))
        return 0

    symbol = sys.argv[1]
    interval = sys.argv[2]
    range_ = sys.argv[3]

    if not range_ in [
        "1d",
        "5d",
        "1mo",
        "3mo",
        "6mo",
        "1y",
        "2y",
        "5y",
        "10y",
        "ytd",
        "max",
    ]:
        print("Error: Invalid range")
        return -1

    params = {
        "region": "US",
        "lang": "en-US",
        "includePrePost": False,
        "interval": interval,
        "useYfid": True,
        "range": range_,
        "corsDomain": "finance.yahoo.com",
        ".tsrc": "finance",
    }

    url = "https://query1.finance.yahoo.com/v8/finance/chart/" + symbol

    resp = requests.get(
        url,
        params=params,
        headers={
            "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36"
        },
    )
    print(resp.url)

    result_dict = resp.json().get("chart", dict()).get("result")[0]
    timestamps = result_dict.get("timestamp")
    quote_dict = result_dict.get("indicators").get("quote")[0]
    open_prices = quote_dict.get("open")
    close_prices = quote_dict.get("close")
    high_prices = quote_dict.get("high")
    low_prices = quote_dict.get("low")

    out_f = open(symbol + ".csv", "w", encoding="utf-8")

    csv_writer = csv.DictWriter(out_f, fieldnames=FIELDNAMES, lineterminator="\n")
    csv_writer.writeheader()

    for i in range(len(timestamps)):
        row = {
            "symbol": symbol,
            "timestamp": timestamps[i],
            "open": open_prices[i],
            "close": close_prices[i],
            "high": high_prices[i],
            "low": low_prices[i],
            "url": resp.url,
        }

        csv_writer.writerow(row)

    out_f.close()


if __name__ == "__main__":
    main()

We went through a couple of simple Yahoo! Finance scraping examples, but there’s far more data that could be scraped from there. One could scrape financial news articles to do market sentiment analysis on them, further quantitive data from Statistic, Historical Data, Financials and other tabs of stock page. For some markets, real time data is available via web socket.

Furthermore, it is possible to do pure API scraping by tapping into Yahoo! Finance mobile app API communications and reproducing them programmatically. This would let us avoid doing any HTML parsing with XPath queries, like we avoided in our time series example.

Vim is touch-typing on steroids

Thu, 12 Jan 2023 00:00:00 +0000

Introduction

I don’t suppose I need to do much explaining on the value of touch typing for developer productivity. Today developers gain additional productivity by using Integrated Development Environments such as PyCharm, Xcode, Visual Studio that provide features like auto-completion, enhanced code navigation, integration with compilers, debuggers and static analysers. All of that does not come for free. Running heavy GUI environment can be resource intensive. For example, Android Studio is quite infamous for making laptop fans spin like crazy. Furthermore, big GUI apps can potentially have quite a few bugs due to their size and complexity. Before Android Studio became an established environment to develop Android apps it was not uncommon for Android app developers to waste a lot of time fighting the bloated, unstable mess that is (was?) Eclipse IDE. Lastly, large GUI apps can be pain to run in server environment. The general, philosophical answer to all of these issues is to seek minimalism in your tooling to the extent it is possible. It may not be very feasible to develop an iOS or Android app with no GUI by merely using CLI tools, but if you’re the intended reader of this blog you can easily do web scraper development by writing and running all your code in terminal environment without any GUI tools other than web browser.

The contents of this post are largely based on Mastering Vim course by Damian Conway, a prominent open source figure. As I watched the course, I took notes that hopefully will be useful to other developers as well. Do not be turned off by the oldness of source material here. Even though the course was released back in 2014 the Vim editor is one of the timeless classics in the realm of Unix tooling. Many things one could learn about Vim are still applicable today and will most likely remain applicable in year 2034.

Despite being primarily a Text User Interface (TUI) program, Vim is quite complex and has literally thousands of commands. One does not have to know all of them to be a productive Vim user. Furthermore, Vim has an extensive ecosystem of plugins, to the extent that Vim can be extended almost into quite complex IDE. We will not be getting much into that, as the current objective is to learn the common denominator of Vim usage and develop a skill in using Vim in it’s pure form, so that if you connect via SSH into some remote Linux/Unix server for the first time you would know exactly how to do text editing there.

There are three primary modes in Vim. In normal (command) mode Vim accepts commands. In insert mode it let’s you type and edit text. In visual mode it allows the user to visually highlight text and do some bulk editing operations.

So let us get started. As you launch Vim, you will be in normal mode by default, until you switch to some other mode. The following things are applicable in the normal mode.

Normal mode commands

To access Vim documentation system, type :help. This will let you browse the docs. To search the docs, you can use :help <keyword>. If you’re unsure of the keyword, you can type some of the keyword and use Tab key for a list of possible completions. Furthermore, one can search the documentation by regular expression with :helpgrep command, e.g. :helpgrep lookahead - no slashes needed. Use :cnext and :cprev commands to go next and previous search result. If you want to skip ahead to next file, use :cnfile command. To go to previous file, use :cpfile command.

More generally, one can use :vimgrep command to search for regex in files other than Vim docs, such as your codebase. It takes a regex (with slashes this time) and one or more file names.

When using Vim help system, you will find that some words or phrases are colored differently and/or delineated by pipe characters. These are hyperlinks that can be traversed by positioning your cursor within the bars and pressing Ctrl-]. To go back one level, one can use Ctrl-T. To get out of help system, one can use :q command or type ZZ.

One can save a lot of time and keystrokes by learning how to navigate text efficiently. One can navigate text in normal mode one character at a time by using arrow keys or h for left, l for right, k for up and l for down. Furthermore, there are ways to perform higher-order motions across words, lines and greater “text objects”. In some systems text can also be navigated vertically by mouse wheel or touch pad gestures.

To find out where you are in the file one can use Ctrl-G. This will give you something like the following line at the bottom of your Vim UI:

"vim-is-touch-typing-on-steroids.md" [Modified] line 64 of 66 --96%-- col 58

We have file name, file state (modified with unsaved changes), current line, number of total lines, number of current column and what percentage the current line number is from the total. Furthermore, you can put the following line in your .vimrc file:

set ruler

This will enable slightly simpler position indication with just line and column numbers.

w moves to start of next word, b - to start of previous word, e - to end of next word and ge to end of previous word. That accelerates your text navigation as you can use single keystroke to move across word, not just a character. But what is a word? In Vim terminology, word is any sequence of characters delimited by non-identifier chars (e.g. whitespace, comma). However, there’s also capital-W Words in Vim that are delimited by whitespace. So we also have W command that moves to next capital-W Word and also B, E and gE commands that perform equivalents to aforementioned word-level motions.

Furthemore, there are line and paragraph motion commands. 0 or | takes you to the start of current line, ^ to start of first word on current line, $ to end of the current line, Enter to start of next line and - to start of previous line. One can also use { to move to start of current paragraph (a continuous cluster of non-empty lines) and } to end of current paragraph. This will position the cursor to the empty line just before or just after a paragraph. If these commands are used on empty lines Vim will move the cursor accordingly to previous or next empty line.

To move to the top of the buffer, one can use gg. To move to the end - G. To move to n-th line, type the line number in normal mode and go G or gg. To move to a certain percentage across the buffer, one can type number and percent sign. For example, 50% will move your cursor halfway through the buffer. However if you just use single % command it will bounce you between some matching brackets ({ ... }, ( ... ), [ ... ]) in your code.

One can also type a number in normal mode before a motion to do that motion a given number of times. lllll can be done faster as 5l and so on. This works on all motions commands and many oter commands as well.

To find the next instance of word under cursor, type \*. For previous instance, type #. You can use :set ignorecase command to make Vim search in case-insensitive manner. After that is on, you can do :set smartcase command which will let you search in case-sensitive way on as-needed basis. For example /selector would search in case-insensitive way, but /Selector would be a case-sensitive search.

You can turn incremental search by launching command :set incsearch. It searches text as you type it and highlights it for you (but it only moves the cursor when you hit Enter; you can hit Escape to back off from the search).

To make Vim highlight all the search results within the current buffer one can turn on search highlighting by doing :set hlsearch. To get rid of the highlighting afterwards one can do a :nohlsearch command.

We can also run a command like :s/<pattern>/<replacement>/ to find a pattern (could be regular expression) and replace it with another pattern within the current line. There are some variations to this command:

:10,33s/from/to/ - apply search and substitution across a specific range of lines (e.g. from 10th to 33rd line).
:1s/from/to/ - do it on a specific line with given line number
-10,+33s/from/to/ - line range can also be relative to the current line number
-10;+33s/from/to/ - if you use semicolon between numbers, the second number will be relative not to the current line number, but to the first line number in the range
Metacharacters can also be used in ranges: . means first line, $ means the last one and so forth.
:%s/from/to/ (or :1,$s/from/to) does the substitution across the entire file.

You can also use patterns to specify ranges based on text content. For example, /<pat>/ in the beginning of command tells Vim to do substitution on next line that matches the given pattern. ?<pat> tells it to work on only the previous line that matches a pattern. This can be combined to narrow down the substitution only to specific part of source code in the file being edited (e.g. HTML body). This also works with offsets. For example: ?foo?+1,$-10s/<pattern>/<replacement>/ does the subsitutions from the line after previous instance of foo to 10 lines before end of file.

The above commands will only do the substitution once, but typically you want it done multiple times (e.g. across the entire file). To achieve this, append the /g (global) modifier, e.g. s/from/to/g. If you’re unsure about replacing stuff all the time, you may want to use /c modifier, e.g. %s/cat/feline/gc. This will let you use an interactive flow of approving or rejecting every substitution.

To repeat the last substitution one more time, you can press & or :s and Enter. To repeat it globally, use %s<Enter>.

Beyond search-and-replace, many Vim commands also take ranges and support the above range syntax. Thus it is valuable to learn them in order to be productive with Vim.

Sometimes you want to match lines according to some regex or pattern across the entire file, but want to do something else than replacing text. This can be done with :global (or :g) command. It lets you match some lines according to a pattern and run another colon command across the matching lines. Some examples:

:g /^\s/ :center - centering only the lines that are already indented (start with a space character).
:g! /^\s/ :center - center only lines that are NOT indented.
:g /<ISBN>/ :normal gUU - run a normal mode command (convert all lines containing <ISBN> to upper-case).
:g /./ :.;/^$/join - convert all paragraphs into a single line to make the text less messy if it was copy-pasted in MS Word or something. Note that we use a range .;/^$/ in a colon command being invoked:
- . - match non-empty line
- ; - make the second part of the range relative to first one
- /^$/ - regex to match empty line, closing the range to only have non-empty lines from the start of paragraph.
:g /./ :.;/^\$/-1join - same as above, but leaves the empty lines for readability.

This is quite powerful and can support various advanced text editing use cases.

Yanking is Vim term for copying some text and putting means pasting the text. yy or Y yanks the current line. yw and y} yanks a word and paragraph respectively (from the current position). More generally, y can be followed by any valid motion. For example, y$ copies text until end of line and y/__END__ copies it until first occurence of give pattern (__END__ in this case). This works well with incremental search, as you would get an indication of where the copied part would end.

We can also use text object specification with this command. For example, to copy the entire word under the cursor, we say yaw (or yaW for capital-W Word). Likewise, yas copies entire sentence and yap does the same for entire paragraph that the cursor is in.

For code editing, Vim lets you yank delimited portions of it:

yab or ya{ for curly-brace delimited block
ya[ for [ ... ]
ya( for ( ... )
ya< for < ... >
ya" for double-quoted text
ya' for single-quoted text

p command is for pasting (putting) the text that was yanked.

To undo the last buffer change, use u command. To re-do it, use Ctrl-R. These can be done multiple times. Think of it as timeline of changes. Since Vim version 7 this timeline can be branched. g- would switch to previous branch and g+ switches to next version of history.

Furthemore, one can “time-travel” in the text change history by using :earlier and :later commands with time offset. For example, :earlier 10m takes you to buffer state that was 10 minutes ago and :later 30s fast-forwards it to what it was 30 seconds later from that time. So if you remember that your code was fine 10 minutes ago, you can go back to what it was back then and either discard the changes you did or move a bit forward in the history to keep some of your work.

However, all of this change is history is discarded when you quit Vim. However, Vim 7.3 introduced a feature of persistent undo history that can be enabled by putting the following lines in your .vimrc file:

if has('persistent_undo')
    set undofile
endif

This will make Vim create a history file in each directory being edited. To consolidate all the edit histories in a single directory somewhere, set undodir in your .vimrc. By default, Vim will remember 1000 levels of changes. You can set undolevel to customise this amount.

Vim also keeps track of you command history. If you want to rerun a colon command you can type : and use up and down keys to navigate the history until you find the command that you want to repeat. You can also type the initial few characters of your command to narrow it down. Furthermore, you can use autocompletion with Tab key when typing your colon commands. This works with commands, file names, command arguments and so forth - pretty much all Vim knows about can be autocompleted.

To cut/delete the current line you can do dd or :d.

To write the contents of buffer to file, use :w command (optionally with file name). To open another file, use :edit command. To quit Vim, use :q (or :q! if you want to discard changes). Writing out and quitting can be done in a single command: :wq or :x.

Insert mode commands

Ctrl-Y duplicates whats in the same column on the preceding line (one character at a time). Ctrl-E does this from the next line. Ctrl-A inserts the last inserted text again. Ctrl-R= evaluates any valid Vimscript expression (can be simple arithmetic) and insert the result. For example, one could type 2+2 in the prompt at the bottom, press Enter and 4 would be inserted.

Ctrl-T increases code indentation level by inserting a single Tab at the start of the line without moving the cursor. Ctrl-D is the opposite - it removes a single tab from start of current line. Ctrl-W deletes one word that precedes the cursor. Ctrl-O takes you to the normal mode for one command.

Vim dialect of regular expressions is similar to these of sed(1) and Perl, but has some key differences. For example, nearly all metasyntactic characters require escaping them with a backslash.

Typing everything in it’s entirety is quite a pain and one of major arguments for using an IDE. However, can also do auto-completion in insert mode. One could type some of the word and press Ctrl-X to enter completion. Next, you use another key combination to indicate what kind of completion you want. For example, Ctrl-X Ctrl-F is for autocompleting file names and file paths. Doing this will present a pop-up with all possible completions that you can choose from.

Ctrl-X Ctrl-D completes a C preprocessor macro by default, but you can set define to any regular expression to use it for other things. But that’s a bit of pain.

Ctrl-X Ctrl-N does identifier completion that works across all sequences of keyword characters across the current edit session. One can edit iskeyword option to change what it considers a keyword character.

Ctrl-X Ctrl-I is for C and C++ programming. It works like the regular identifier completion, but also goes into source code files that were #included.

Ctrl-X Ctrl-O is a smart, unified “omni-completion” that tries to use some heuristics to present some options to you in a smart, automatic way that is informed by the context of where you are in the code that you are editing. This typically requires a completion specification (some are included in the standard installation - see :help compl-omni-filetypes). Furthermore, one may need to preprocess the source code with a tool called Exuberant ctags. You may also want to enable file type detection by doing :filetype plugin on.

Visual modes

When you type v Vim enters the visual mode. As you move your cursor around, it will hightlight the text and select it. Once you have selected the exact text you want, you can run a normal mode command and it will apply to that portion of text. That is the basic visual mode.

There’s also visual line mode that entails selecting entire lines. You can enter it by typing capital V in the normal mode (or in basic visual mode if you want to switch).

Lastly, there’s a visual block mode for selecting a rectangular block to deal with something like ASCII art tables. It can be entered by typing Ctrl-V.

Scrapy simplified: developing a single file web scraper

Mon, 26 Dec 2022 00:00:00 +0000

The default way of using Scrapy entails running scrapy startproject to generate a bunch of starter code across multiple files. For small scale scrapers that’s a bit of overkill. It’s far simpler to have a single Python script file that you can run when you want to scrape some data. The CrawlerProcess class in Scrapy framework enables us to develop such a script.

For the sake of example, let us suppose we are interested in extracting some basic information about Fortune 500 companies from the Fortune 500 website. To do a little exploration and planning, let us load the Fortune 500 search page via Scrapy shell:

$ scrapy view "https://fortune.com/ranking/fortune500/2022/search/"

This opens up a browser window with HTML as the Scrapy was able to fetch it. Luckily we did not run into any countermeasures against scraping. If we view the page source we find that all the info being rendered in the table is available in JSON format inside <script> with id equal to __NEXT_DATA__. This JSON string is the biggest thing in the HTML document that Scrapy fetched. So that’s what we’re going to scrape to get links to company pages.

Screenshot 1

Let us look into a company profile page.

$ scrapy view "https://fortune.com/company/verizon/fortune500/"

Once again, we find the same kind of script tag from Next.js framework that we can extract data from.

Screenshot 2

Let us go through the following example scraper.

#!/usr/bin/python3

import json

import scrapy
from scrapy.crawler import CrawlerProcess
from scrapy.selector import Selector

class F500Spider(scrapy.Spider):
    start_urls = ["https://fortune.com/ranking/fortune500/2022/search/"]
    name = "f500"

    def start_requests(self):
        yield scrapy.Request(self.start_urls[0], callback=self.parse_search)

    def parse_search(self, response):
        json_str = response.xpath('//script[@id="__NEXT_DATA__"]/text()').get()
        json_dict = json.loads(json_str)

        item_dicts = (
            json_dict.get("props", dict())
            .get("pageProps", dict())
            .get("franchiseList", dict())
            .get("items", [])
        )

        for item_dict in item_dicts:
            slug = item_dict.get("slug")
            rank = item_dict.get("data", dict()).get("Rank", "").replace(",", "")
            rank = int(rank)

            if slug is not None and rank <= 500:
                yield response.follow(slug, callback=self.parse_company_page)

    def parse_company_page(self, response):
        json_str = response.xpath('//script[@id="__NEXT_DATA__"]/text()').get()
        try:
            json_dict = json.loads(json_str)
        except:
            return
        
        fli = json_dict.get("props", dict()).get("pageProps", dict()).get("franchiseListItem")
        if fli is None:
            return

        company_name = fli.get("title")
        rank = fli.get("rank")

        company_info = fli.get("companyInfo", dict())

        website = company_info.get("Website")
        if website is not None and website.startswith("<a"):
            website = Selector(text=website).xpath('//a/text()').get()
    
        country = company_info.get("Country")
        industry = company_info.get("Industry")
        ticker = company_info.get("Ticker")

        yield {
            "rank": rank,
            "company_name": company_name,
            "website": website,
            "country": country,
            "industry": industry,
            "ticker": ticker
        }


def main():
    # See: https://docs.scrapy.org/en/latest/topics/practices.html#run-from-script
    process = CrawlerProcess(settings={
        "FEEDS": {
            "f500.csv": {"format": "csv"}
        }
    })
    process.crawl(F500Spider)
    process.start()


if __name__ == "__main__":
    main()

The key part is the spider that we develop by creating a subclass of scrapy.Spider, the same way we would be doing in spiders/ directory of regular Scrapy project. To launch the spider without the entire shebang that would come with Scrapy project we need to instantiate a scrapy.crawler.CrawlerProcess object with settings dictionary, call the crawl() method with our spider class and the call the start() method. This will make the spider run until completion. Script will finish running once the scraping is done.

If you run this you may notice that 498 rows are saved to f500.csv, not 500. That’s because two of the company pages are implemented differently and have their data hard-coded in JavaScript snippets. Since the objective here was to demonstrate how to use the crawler process for developing a simplified scraper, I did not proceed with implementing support for scraping these two pages, although that could be done with using js2xml library.

Getting the data under version control with dolt

Thu, 15 Dec 2022 00:00:00 +0000

Many developers would agree that source code version control is a great thing that they would not imagine the modern software development without. What if it was applied for structured data as well? Yes, technically it is possible to save SQLite file or SQL dump into git repo, but that is rather clunky and outside the intended use case of git. For proper version control, we would want row-level diffing, ability to undo changes, group them into incremental chunks (commits), have a way to review the proposed changes and many other nice features that we have available in systems like git.

Web scraper developers may find data version control to be of interest for certain use cases. For example, one might be doing daily rescrapes of product prices on eCommerce portal for the purpose of price intelligence. Version-controlling the data provides an easy way to query the database for changes between scraping jobs if we commit the data at the end of each scraping job. This would enable automatically diffing the daily snapshots and generating change report to be seen by the client paying for the scraping operation.

Dolt project is an up-and-coming DBMS that is meant to be a hybrid between MySQL and git. It supports an SQL dialect that greatly overlaps with that of MySQL and also provides a version control features that are based on the ones in git.

Dolt is a single-binary Go program that can be downloaded from Github tags page or compiled from source code. It is also available from Homebrew on macOS. Furthermore there are two official Docker images on Docker Hub:

dolthub/dolt - for running dolt as CLI tool.
dolthub/dolt-sql-server - for running it in SQL server mode.

Dolt CLI is quite similar to git CLI:

$ dolt 
Valid commands for dolt are
                init - Create an empty Dolt data repository.
              status - Show the working tree status.
                 add - Add table changes to the list of staged table changes.
                diff - Diff a table.
               reset - Remove table changes from the list of staged table changes.
               clean - Remove untracked tables from working set.
              commit - Record changes to the repository.
                 sql - Run a SQL query against tables in repository.
          sql-server - Start a MySQL-compatible server.
          sql-client - Starts a built-in MySQL client.
                 log - Show commit logs.
              branch - Create, list, edit, delete branches.
            checkout - Checkout a branch or overwrite a table from HEAD.
               merge - Merge a branch.
           conflicts - Commands for viewing and resolving merge conflicts.
         cherry-pick - Apply the changes introduced by an existing commit.
              revert - Undo the changes introduced in a commit.
               clone - Clone from a remote data repository.
               fetch - Update the database from a remote data repository.
                pull - Fetch from a dolt remote data repository and merge.
                push - Push to a dolt remote.
              config - Dolt configuration.
              remote - Manage set of tracked repositories.
              backup - Manage a set of server backups.
               login - Login to a dolt remote host.
               creds - Commands for managing credentials.
                  ls - List tables in the working set.
              schema - Commands for showing and importing table schemas.
               table - Commands for copying, renaming, deleting, and exporting tables.
                 tag - Create, list, delete tags.
               blame - Show what revision and author last modified each row of a table.
         constraints - Commands for handling constraints.
             migrate - Executes a database migration to use the latest Dolt data format.
         read-tables - Fetch table(s) at a specific commit into a new dolt repo
                  gc - Cleans up unreferenced data from the repository.
       filter-branch - Edits the commit history using the provided query.
          merge-base - Find the common ancestor of two commits.
             version - Displays the current Dolt cli version.
                dump - Export all tables in the working set into a file.
                docs - Commands for working with Dolt documents.

We have commands for adding changes to staging area, commiting them to change history, making branches, communicating with remote servers, resolving merge conflicts and so on. Note, however, that not every VCS feature supported by git is supported by dolt. Furthemore, we see some commands related to DB functionality such as sql, schema and dump. Lastly, we have docs command that provides a little of functionality for version-controlling your database documentation, such as README file and such.

Dolt stores all the database contents in a single directory per database, which makes it easy to backup and migrate between servers.

SQL dialect of Dolt is largely the same as the one used in MySQL with two notable differences. First, not all advanced features of MySQL are supported at this point (see the list of supported statements). Second, Dolt introduces some more SQL statements that are related to version control aspect (see Version control section of the official documentation).

To run Dolt in SQL server mode one could run dolt sql-server command. This launches a network server that talks the MySQL protocol and can be used with a MySQL client programs or libraries (e.g. mysql-connector). Furthermore, Dolt includes a simple SQL client that can be launched by running dolt sql-client. There are also some official Python client libraries for Dolt:

DoltPy implements Dolt SQL client
doltcli is a wrapper around dolt CLI tool to run queries locally without running Dolt as server.

For git users, there’s Github for community and collaboration. DoltHub is a company and portal that not only develops dolt, but also lets you publically host your databases. If you don’t want to host it publically there are two options:

DoltLab - a self-hostable, more private solution similar to GitLab. DoltLab is to dolt what GitLab is to Git.
Hosted Dolt - an equivalent of AWS RDS for Dolt server. You get a dolt server that is provisioned and maintained for you, no DBA needed.

DoltHub Inc. is also running public data bounties that entail gathering open data by scraping, wrangling and importing it into structured database. Participants are paid based on how much of the data they submitted via pull requests was accepted. This can be an opportunity to make some money if you’re a web scraper developer.

Understanding SPF, DKIM, DMARC for email security and deliverability

Wed, 14 Dec 2022 00:00:00 +0000

On it’s own SMTP protocol does not do much validation on the authenticity of sender. One can spoof protocol headers at SMTP and MIME levels to send a message in another users name. That can be problematic as it makes spam and social engineering attacks easier. To address this problem, some email authentication technologies have been developed. We will discuss the three major ones: SPF, DKIM and DMARC. SPF and DKIM authenticate sender metadata and DMARC extends them to configure email handling in case authentication fails. All of these rely on DNS records for the sender domain.

SPF: Sender Policy Framework

Sender Policy Framework, specified in RFC 7208 is a way for domain owners to authorize or disallow email sending from email addresses associated with that domain via DNS TXT records. This sets a policy regarding what entities (if any) can use the domain for their sender email identities. If a receiving or intermediate email server is compliant with SPF, it would check the DNS records against source email address in the SMTP HELO/EHLO/MAIL FROM messages and base the decision to allow or reject the message based on the policy that domain owner has published. If a sending domain does not publish any SPF records, the SPF-compliant email server is allowed to drop the email message, which has implications for email deliverability. Setting up SPF records is one of the recommended practices to prepare a domain for sending cold email.

Suppose we have a domain example.org and we want to disallow any and all email from that domain. We would publish the following DNS TXT record:

v=spf1 -all

All SPF records start with version prefix v=spf1 and contains zero or more directives (mechanism-qualifier pairs). It can also contains some optional modifiers.

In this case - is a qualifier that means “fail” (i.e. hard error on sending) and all is a mechanism that matches all sender IP addresses.

On the other, if we wanted everyone to be allowed to use the domain for email sending, we could publish this:

v=spf1 +all

Now there’s a qualifier + that means “pass”, thus email sending is allowed from all IP addresses.

There are two more possible qualifiers. ~ means “softfail”, which corresponds to lesser error condition on sending an email. This may cause email to be marked as possible spam or bounced with a note to retry it later. ? neither allows not disallows email to be sent.

There are the following mechanisms that SPF supports:

all - match any and all sender IP addresses
include - include SPF records from other domain (e.g. include:example.net)
a - match all IPs in domain name A records (e.g. just a for the current domain or a:example.net for another domain)
mx - match all IPs in domain name MX records (can also point to another DNS name)
ptr - match IP addresses that reversely-resolves to given domain name or its subdomain (not recommended)
ip4 - match a single IPv4 address or IPv4 range (e.g. ip4:1.2.3.4 or ip4:192.168.0.1/16)
ip6 - like ip4, but for IPv6 addresses
exists - takes a domain spec and matches iff that matches at least one A record; meant for advanced use cases.

SPF implementations go through directives from left to right until it finds the one that conclusively (dis)allows email transfer, runs into error conditions or runs out of directives to check. In the case of conflicting directives, the one earlier in the records has precedence.

At the end of the SPF record there can be the following modifiers:

redirect - redirect SPF policy checks to another domain
exp - used for configuring error message when email is rejected.

Now that we’re familiar with concepts and some technical details of SPF, let us see how SPF is implemented by a real world email provider - hey.com. We use dig(1) to see what kind of SPF records hey.com domain has published:

$ dig -t TXT hey.com | grep spf1
hey.com.		60	IN	TXT	"v=spf1 include:_spf.hey.com ~all"

There’s two SPF directives here:

include:_spf.hey.com - include SPF policy of subdomain
~all - soft-reject all email that is not covered by any other directive.

The include mechanism triggers recursive check. Now we must check the SPF records of a referenced subdomain:

$ dig -t TXT _spf.hey.com | grep spf1
_spf.hey.com.		120	IN	TXT	"v=spf1 ip4:204.62.114.0/23 -all"

This specifies an IPv4 address range that is allowed to send email messages from mailboxes under hey.com domain and strongly forbids all email traffic to be sent from elsewhere, thus overriding the ~all directive in the previous record.

If you are growth hacker working on cold emails campaigns it is strongly recommended to set up SPF for all your outreach domains. Your email provider will most likely provide the exact instructions on how to do that.

But what if you want to do bug bounties? In that case, SPF records or lack thereof are not that relevant. Missing SPF records are not generally considered to be a serious security problem and reporting that is unlikely to lead to any payout.

DKIM: Domain Keys Identified Mail

DKIM, specified in RFC 6376 and some other additional RFCs goes beyond simple domain-level policies and enables email message signing by public key published in DNS record. This lets the intermediate email server to check if email message was sent from the keeper of respective private key (e.g. sender email provider) and that there was no tampering by adversarial parties (e.g. rogue email servers) as the message was enroute to the recipient. Depending on how exactly DKIM is deployed, the cryptographic signature may cover all message or only some headers.

DKIM also relies on DNS TXT records, but they are required to be published at subdomain of following form: [selector]._domainkey.[domain]. Here selector is provider-specific value and domain is a sender domain.

The DKIM signature is computed by sending email server that is managed by your email provider. As the email message goes through further DKIM-compliant servers it is being verified for the valid signature. DKIM implementation by itself does not reject any messages, but introduces extra email headers that can be used by spam filtering solutions.

DKIM has implications for email deliverability. As far as anti-spam systems are concerned, lack of DKIM record means less reliable sender identity. Furthermore, DKIM misconfiguration can cause signature validation to fail, which can lead to message being filtered as spam (especially if DMARC is being used).

RFC 6367 provides the following example of DKIM-Signature header:

DKIM-Signature: v=1; a=rsa-sha256; d=example.net; s=brisbane;
   c=simple; q=dns/txt; i=@eng.example.net;
   t=1117574938; x=1118006938;
   h=from:to:subject:date;
   z=From:foo@eng.example.net|To:joe@example.com|
    Subject:demo=20run|Date:July=205,=202005=203:44:08=20PM=20-0700;
bh=MTIzNDU2Nzg5MDEyMzQ1Njc4OTAxMjM0NTY3ODkwMTI=;
b=dzdVyOfAKCdLXdJOc9G2q8LoXSlEniSbav+yuU4zGeeruD00lszZVoG4ZHRNiYzR

The following is an example of DKIM record that ZOHO provides:

v=DKIM1; k=rsa; p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDXzlbk53i+yhhnbLM6Z/dzASCoGrRMbnMJzFsJK9Q17cHVDqmSUjQkNMfBhMD0JgwZcuUpHBE2aTVzwaLEQXRkR7BDOLBY+JpGjtJ45sjI9rmlN/4ntJLFEzuGaaHDp/XyR3eORKSt2QBHs9OM9U8zMoXCQc1+MW6vbtYJeWhacwIDAQAB

Like with SPF, your email provider will probably give you exact DNS record to use for enabling DKIM for your domain.

DMARC: Domain-based Message Authentication, Reporting and Conformance

DMARC is something that builds upon SPF and DKIM. It allows domain owners to publish DNS records telling how messages that violate SPF or DKIM authorization should be treated (rejected or quarantined). Furthermore, domain owners can provide an email address for the aggregate or forensic notifications to be sent to, so that they could be used for troubleshooting. Furthermore, you may want to know if someone if trying to spoof your email for malicious purposes.

RFC 7489 is the specification to read if you want to know more about DMARC.

SMTP, IMAP, POP3 and MIME: understanding email protocols

Mon, 21 Nov 2022 00:00:00 +0000

Introduction and big picture

Email is a fairly old school technology meant to replicate postal service digitally. To understand how email works, we must know about network protocols that specify sets of data formats and data exchange rules for exchanging messages over the network. In the modern internet email sending part is conceptually (and sometimes technically) separate from receiving part. Email sending aspects are formalised in RFCs that describe SMTP protocol, whereas reception part can be done either via IMAP or POP3 protocol. The conceptual difference between IMAP and POP3 is that IMAP is meant to provide a way to browse mailboxes via network connection whereas POP3 is meant to download received messages into user-controlled computer and let them read it offline without any persistent connection to the server. Furthermore, MIME data format is used for encoding contents and inner structure of the messages that are transferred.

Broadly speaking, email system is federated and consists of the following parts:

Mail Transfer Agent (MTA) is a program that implements SMTP protocol to transport messages between hosts (e.g. Sendmail, qmail, Postfix).
Mail User Agent (MUA) is the email client application (e.g. mutt, Mail.app, Outlook, Thunderbird).
There can also be a Mail Delivery Agent (MDA) - an intermediate piece of software that bridges the gap between MTA and MUA on the receiving side (e.g. procmail). This can be done for spam filtering purposes and to manage the email message persistence on Unix/Linux systems.
Lastly, there can also be a Mail Submission Agent (MSA) that is equivalent of MDA, but for sending the email.

The general big-picture flow of email message is the following:

A sender creates the email message and sends it from MUA to MTA managed by their email provider (possibly via MSA). SMTP protocol is being used here.
Message traverses one or more MTAs. Again, this is done via SMTP protocol.
Message reaches the final MTA belonging to the email provider of the recipient and is stored there.
Recipient uses MUA on their end to retrieve the message (possibly via MDA). Either IMAP or POP3 protocol can be used here.

It should be noted that email provider specific REST API can be used between MUA and MTA (possibly through a client library). For examples of such APIs, see:

This would replace SMTP, IMAP and POP3 protocols for communications between client and server.

SMTP: Simple Mail Transfer Protocol

SMTP was originally formalised back in 1981 in RFC 788. Since then, the protocol specification has been extended across various RFCs to keep it up to data with the changing needs of growing internet. At the time of writing, RFC 5321 is the final version of core SMTP spec. SMTP is a text-based protocol with two kinds of messages: commands and responses.

Let us use smtplib module from the vanilla Python installation to explore the SMTP debugging output. We want to see the protocol flow.

$ python3
Python 3.10.8 (main, Oct 13 2022, 09:48:40) [Clang 14.0.0 (clang-1400.0.29.102)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import smtplib
>>> smtp_conn = smtplib.SMTP("smtp.rambler.ru", port=587)
>>> smtp_conn.set_debuglevel(1)
>>> smtp_conn.starttls()
send: 'ehlo [[REDACTED]]\r\n'
reply: b'250-mail.rambler.ru\r\n'
reply: b'250-SIZE 104857600\r\n'
reply: b'250-ENHANCEDSTATUSCODES\r\n'
reply: b'250-8BITMIME\r\n'
reply: b'250-AUTH PLAIN LOGIN\r\n'
reply: b'250 STARTTLS\r\n'
reply: retcode (250); Msg: b'mail.rambler.ru\nSIZE 104857600\nENHANCEDSTATUSCODES\n8BITMIME\nAUTH PLAIN LOGIN\nSTARTTLS'
send: 'STARTTLS\r\n'
reply: b'220 2.0.0 Start TLS\r\n'
reply: retcode (220); Msg: b'2.0.0 Start TLS'
(220, b'2.0.0 Start TLS')

We established TCP connection to SMTP server and sent EHLO command with our IP address to initiate the conversation. The server responded with status code 250 (indicating success) and a list of SMTP service extensions that server is supporting. To make the connection encrypted, we also issued a STARTTLS command. To this command server replied with status code 220, which means that it ready to serve the client.

Now we need to login to the server with account credentials.

>>> username = "[REDACTED]"
>>> password = "[REDACTED]"
>>> smtp_conn.login(username, password)
send: 'ehlo [202.3.218.138]\r\n'
reply: b'250-mail.rambler.ru\r\n'
reply: b'250-SIZE 104857600\r\n'
reply: b'250-ENHANCEDSTATUSCODES\r\n'
reply: b'250-8BITMIME\r\n'
reply: b'250 AUTH PLAIN LOGIN\r\n'
reply: retcode (250); Msg: b'mail.rambler.ru\nSIZE 104857600\nENHANCEDSTATUSCODES\n8BITMIME\nAUTH PLAIN LOGIN'
send: 'AUTH PLAIN [REDACTED]\r\n'
reply: b'235 2.0.0 OK\r\n'
reply: retcode (235); Msg: b'2.0.0 OK'
(235, b'2.0.0 OK')

Calling login() method repeated the EHLO message over the now-encrypted channel and sent AUTH message that deals with “PLAIN Simple Authentication and Security Layer (SASL) Mechanism” described in RFC 4616. The redacted part is Base64 string from the following sequence:

Null byte (0x00).
Email account username.
Null byte.
Email account password.

We got response from the server saying that all is good with the login step. Now we can send some test message.

>>> to_addr = "noreply@keyspace.lt"
>>> smtp_conn.sendmail(username, to_addr, "Test!")
send: 'mail FROM:<[REDACTED]> size=5\r\n'
reply: b'250 2.1.0 Ok\r\n'
reply: retcode (250); Msg: b'2.1.0 Ok'
send: 'rcpt TO:<noreply@keyspace.lt>\r\n'
reply: b'250 2.1.5 Ok\r\n'
reply: retcode (250); Msg: b'2.1.5 Ok'
send: 'data\r\n'
reply: b'354 End data with <CR><LF>.<CR><LF>\r\n'
reply: retcode (354); Msg: b'End data with <CR><LF>.<CR><LF>'
data: (354, b'End data with <CR><LF>.<CR><LF>')
send: b'Test!\r\n.\r\n'
reply: b'250 2.0.0 Ok: queued as CB14F2642E52\r\n'
reply: retcode (250); Msg: b'2.0.0 Ok: queued as CB14F2642E52'
data: (250, b'2.0.0 Ok: queued as CB14F2642E52')
{}

There a sequence of three steps here:

We provide the email address of the sender which will correspond to email account username in most cases via MAIL FROM command.
We provide recipient address via RCPT TO command.
We send DATA command, contents of the message and message end sequence. In this case the message is a very simple ASCII string, but in real world this is typically a MIME payload that will be discussed later.

All step succeeded and a message was enqueued for further transfer. To finalise the connection in a clean way we would send a QUIT command.

We only covered the happy path here. RFC 5321 Appendix D lists more examples.

Futhermore, modern SMTP implementation deal with additional considerations, such as spam filtering and policy enforcement.

Standard TCP port of plaintext SMTP is 25. SMTP-over-TLS typically uses port 587.

One might ask: given an email address, how does MTA know the IP address of the next MTA to pass the message to? This is typically found through DNS MX records that point to hostname of the mail server servicing the given domain name.

IMAP: Internet Message Access Protocol

IMAPv4rev2 is the current version of IMAP protocol, specified in RFC 9501. This is fairly new standard that is not always implemented to the letter. However, IMAP protocol has been around for long time and is widely supported in the world of network software. Just like SMTP, IMAP is text-based, line-oriented protocol.

Let us receive an email with imaplib Python module and go through IMAP protocol flow. We will enable debug output in Python REPL and reproduce the steps that email client would do when receiving messages.

Like with SMTP protocol, we first need to establish the connection and login to an account.

>>> import imaplib
>>> imap_conn = imaplib.IMAP4_SSL("imap.rambler.ru", port=993)
>>> imap_conn.debug = 5
>>> imap_conn.login(username, password)
  33:33.96 > b'OGCE1 LOGIN [REDACTED] "[REDACTED]"'
  33:34.54 < b'OGCE1 OK [CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS BINARY MOVE SNIPPET=FUZZY PREVIEW=FUZZY PREVIEW STATUS=SIZE SAVEDATE LITERAL+ NOTIFY SPECIAL-USE QUOTA] Logged in'
  33:34.54 	matched b'(?P<tag>OGCE\\d+) (?P<type>[A-Z]+) (?P<data>.*)' => (b'OGCE1', b'OK', b'[CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS BINARY MOVE SNIPPET=FUZZY PREVIEW=FUZZY PREVIEW STATUS=SIZE SAVEDATE LITERAL+ NOTIFY SPECIAL-USE QUOTA] Logged in')
  33:34.54 	matched b'\\[(?P<type>[A-Z-]+)( (?P<data>.*))?\\]' => (b'CAPABILITY', b' IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS BINARY MOVE SNIPPET=FUZZY PREVIEW=FUZZY PREVIEW STATUS=SIZE SAVEDATE LITERAL+ NOTIFY SPECIAL-USE QUOTA', b'IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS BINARY MOVE SNIPPET=FUZZY PREVIEW=FUZZY PREVIEW STATUS=SIZE SAVEDATE LITERAL+ NOTIFY SPECIAL-USE QUOTA')
  33:34.54 untagged_responses[CAPABILITY] 0 += ["b'IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS BINARY MOVE SNIPPET=FUZZY PREVIEW=FUZZY PREVIEW STATUS=SIZE SAVEDATE LITERAL+ NOTIFY SPECIAL-USE QUOTA'"]
('OK', [b'[CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS BINARY MOVE SNIPPET=FUZZY PREVIEW=FUZZY PREVIEW STATUS=SIZE SAVEDATE LITERAL+ NOTIFY SPECIAL-USE QUOTA] Logged in'])

Unlike with SMTP connection, we establish TLS connection to server first and start sending IMAP commands in the encrypted channel. IMAP protocol does not have any equivalent of STARTTLS message to upgrade plaintext connection to an encrypted one.

LOGIN command was sent with username and password. Server responded with status OK and a list of supported capabilities.

Now we need to select a mailbox on the server. If we provide no mailbox name the Python IMAP client will switch to inbox by default.

>>> imap_conn.select()
  34:03.21 > b'OGCE2 SELECT INBOX'
  34:03.72 < b'* FLAGS (\\Answered \\Flagged \\Deleted \\Seen \\Draft)'
  34:03.72 	matched b'\\* (?P<type>[A-Z-]+)( (?P<data>.*))?' => (b'FLAGS', b' (\\Answered \\Flagged \\Deleted \\Seen \\Draft)', b'(\\Answered \\Flagged \\Deleted \\Seen \\Draft)')
  34:03.72 untagged_responses[FLAGS] 0 += ["b'(\\Answered \\Flagged \\Deleted \\Seen \\Draft)'"]
  34:03.72 < b'* OK [PERMANENTFLAGS (\\Answered \\Flagged \\Deleted \\Seen \\Draft \\*)] Flags permitted.'
  34:03.72 	matched b'\\* (?P<type>[A-Z-]+)( (?P<data>.*))?' => (b'OK', b' [PERMANENTFLAGS (\\Answered \\Flagged \\Deleted \\Seen \\Draft \\*)] Flags permitted.', b'[PERMANENTFLAGS (\\Answered \\Flagged \\Deleted \\Seen \\Draft \\*)] Flags permitted.')
  34:03.72 untagged_responses[OK] 0 += ["b'[PERMANENTFLAGS (\\Answered \\Flagged \\Deleted \\Seen \\Draft \\*)] Flags permitted.'"]
  34:03.72 	matched b'\\[(?P<type>[A-Z-]+)( (?P<data>.*))?\\]' => (b'PERMANENTFLAGS', b' (\\Answered \\Flagged \\Deleted \\Seen \\Draft \\*)', b'(\\Answered \\Flagged \\Deleted \\Seen \\Draft \\*)')
  34:03.72 untagged_responses[PERMANENTFLAGS] 0 += ["b'(\\Answered \\Flagged \\Deleted \\Seen \\Draft \\*)'"]
  34:03.72 < b'* 10 EXISTS'
  34:03.72 	matched b'\\* (?P<data>\\d+) (?P<type>[A-Z-]+)( (?P<data2>.*))?' => (b'10', b'EXISTS', None, None)
  34:03.72 untagged_responses[EXISTS] 0 += ["b'10'"]
  34:03.72 < b'* 4 RECENT'
  34:03.72 	matched b'\\* (?P<data>\\d+) (?P<type>[A-Z-]+)( (?P<data2>.*))?' => (b'4', b'RECENT', None, None)
  34:03.72 untagged_responses[RECENT] 0 += ["b'4'"]
  34:03.72 < b'* OK [UNSEEN 7] First unseen.'
  34:03.72 	matched b'\\* (?P<type>[A-Z-]+)( (?P<data>.*))?' => (b'OK', b' [UNSEEN 7] First unseen.', b'[UNSEEN 7] First unseen.')
  34:03.72 untagged_responses[OK] 1 += ["b'[UNSEEN 7] First unseen.'"]
  34:03.72 	matched b'\\[(?P<type>[A-Z-]+)( (?P<data>.*))?\\]' => (b'UNSEEN', b' 7', b'7')
  34:03.72 untagged_responses[UNSEEN] 0 += ["b'7'"]
  34:03.72 < b'* OK [UIDVALIDITY 1644072125] UIDs valid'
  34:03.72 	matched b'\\* (?P<type>[A-Z-]+)( (?P<data>.*))?' => (b'OK', b' [UIDVALIDITY 1644072125] UIDs valid', b'[UIDVALIDITY 1644072125] UIDs valid')
  34:03.72 untagged_responses[OK] 2 += ["b'[UIDVALIDITY 1644072125] UIDs valid'"]
  34:03.72 	matched b'\\[(?P<type>[A-Z-]+)( (?P<data>.*))?\\]' => (b'UIDVALIDITY', b' 1644072125', b'1644072125')
  34:03.72 untagged_responses[UIDVALIDITY] 0 += ["b'1644072125'"]
  34:03.72 < b'* OK [UIDNEXT 11] Predicted next UID'
  34:03.72 	matched b'\\* (?P<type>[A-Z-]+)( (?P<data>.*))?' => (b'OK', b' [UIDNEXT 11] Predicted next UID', b'[UIDNEXT 11] Predicted next UID')
  34:03.72 untagged_responses[OK] 3 += ["b'[UIDNEXT 11] Predicted next UID'"]
  34:03.72 	matched b'\\[(?P<type>[A-Z-]+)( (?P<data>.*))?\\]' => (b'UIDNEXT', b' 11', b'11')
  34:03.72 untagged_responses[UIDNEXT] 0 += ["b'11'"]
  34:03.72 < b'* OK [HIGHESTMODSEQ 17] Highest'
  34:03.72 	matched b'\\* (?P<type>[A-Z-]+)( (?P<data>.*))?' => (b'OK', b' [HIGHESTMODSEQ 17] Highest', b'[HIGHESTMODSEQ 17] Highest')
  34:03.72 untagged_responses[OK] 4 += ["b'[HIGHESTMODSEQ 17] Highest'"]
  34:03.72 	matched b'\\[(?P<type>[A-Z-]+)( (?P<data>.*))?\\]' => (b'HIGHESTMODSEQ', b' 17', b'17')
  34:03.72 untagged_responses[HIGHESTMODSEQ] 0 += ["b'17'"]
  34:03.72 < b'OGCE2 OK [READ-WRITE] Select completed (0.009 + 0.000 + 0.008 secs).'
  34:03.72 	matched b'(?P<tag>OGCE\\d+) (?P<type>[A-Z]+) (?P<data>.*)' => (b'OGCE2', b'OK', b'[READ-WRITE] Select completed (0.009 + 0.000 + 0.008 secs).')
  34:03.72 	matched b'\\[(?P<type>[A-Z-]+)( (?P<data>.*))?\\]' => (b'READ-WRITE', None, None)
  34:03.72 untagged_responses[READ-WRITE] 0 += ["b''"]
('OK', [b'10'])

You may notice that first command message was prefixed by OGCE1 and next one by OGCE2. This is IMAP message tagging in action. They are meant to associate some of the reply messages with a command message, as is seen in one of the lines that client got in reponse to SELECT message.

We got several lines in response that mostly deal with message UIDs - 32-bit numbers that uniquely identify messages within the email account.

* FLAGS (\\Answered \\Flagged \\Deleted \\Seen \\Draft) - a list of flags that can be assigned to messages (answered, seen, deleted, etc.)
* 10 EXISTS - number of messages in the mailbox (10).
* 4 RECENT - number of messages with “recent” flag set (4) that recently arrived to mailbox.
* OK [UNSEEN 7] - UID of the first unseen message.
* OK [UIDNEXT 11] - next UID that would be assigned on newly arrived message (predicted value).

Now we can search within selected mailbox for messages matching some predicate (e.g. sender address being equal to known value). For the purpose of this example, we will retrieve a list of UIDs of all messages in the inbox.

>>> imap_conn.search(None, 'ALL')
  34:42.42 > b'OGCE3 SEARCH ALL'
  34:42.73 < b'* SEARCH 1 2 3 4 5 6 7 8 9 10'
  34:42.73 	matched b'\\* (?P<type>[A-Z-]+)( (?P<data>.*))?' => (b'SEARCH', b' 1 2 3 4 5 6 7 8 9 10', b'1 2 3 4 5 6 7 8 9 10')
  34:42.73 untagged_responses[SEARCH] 0 += ["b'1 2 3 4 5 6 7 8 9 10'"]
  34:42.73 < b'OGCE3 OK Search completed (0.001 + 0.000 secs).'
  34:42.73 	matched b'(?P<tag>OGCE\\d+) (?P<type>[A-Z]+) (?P<data>.*)' => (b'OGCE3', b'OK', b'Search completed (0.001 + 0.000 secs).')
  34:42.73 untagged_responses[SEARCH] => [b'1 2 3 4 5 6 7 8 9 10']
('OK', [b'1 2 3 4 5 6 7 8 9 10'])

We got a list of UIDs from 1 to 10. To get a single message, we can send a FETCH command (response removed for the sake of brevity):

>>> imap_conn.fetch(b'10', '(RFC822)')
  35:08.76 > b'OGCE5 FETCH 10 (RFC822)'
  35:09.15 < b'* 10 FETCH (FLAGS (\\Seen \\Recent) RFC822 {2940}'
  35:09.15 	matched b'\\* (?P<data>\\d+) (?P<type>[A-Z-]+)( (?P<data2>.*))?' => (b'10', b'FETCH', b' (FLAGS (\\Seen \\Recent) RFC822 {2940}', b'(FLAGS (\\Seen \\Recent) RFC822 {2940}')
  35:09.15 	matched b'.*{(?P<size>\\d+)}$' => (b'2940',)

In the response we get a MIME payload with all the headers.

Finally, we close the connection and logout:

>>> imap_conn.close()
  36:17.68 > b'OGCE7 CLOSE'
  36:17.96 < b'OGCE7 OK Close completed (0.001 + 0.000 secs).'
  36:17.96 	matched b'(?P<tag>OGCE\\d+) (?P<type>[A-Z]+) (?P<data>.*)' => (b'OGCE7', b'OK', b'Close completed (0.001 + 0.000 secs).')
('OK', [b'Close completed (0.001 + 0.000 secs).'])
>>> imap_conn.logout()
  36:22.24 > b'OGCE8 LOGOUT'
  36:22.57 < b'* BYE Logging out'
  36:22.57 	matched b'\\* (?P<type>[A-Z-]+)( (?P<data>.*))?' => (b'BYE', b' Logging out', b'Logging out')
  36:22.57 untagged_responses[BYE] 0 += ["b'Logging out'"]
  36:22.57 BYE response: b'Logging out'
('BYE', [b'Logging out'])

POP3: Post Office Protocol version 3

POP3 is a network protocol that enables end users to download email messages for offline reading and archiving. Python supports it via poplib module. Like SMTP, POP3 supports both using TLS from start of connection and upgrading the plaintext connection before login step to POP3-over-TLS.

Overall POP3 is much simpler protocol than IMAP. Listing messages and retrieving them one at a time is fairly simple:

>>> import poplib
>>> pop_conn = poplib.POP3("pop3.rambler.ru")
>>> pop_conn.user(username)
b'+OK'
>>> pop_conn.pass_(password)
b'+OK Logged in.'
>>> pop_conn.list()
(b'+OK 10 messages:', [b'1 18561', b'2 18978', b'3 58337', b'4 47221', b'5 48119', b'6 47562', b'7 66608', b'8 71147', b'9 32400', b'10 2940'], 90)
>>> pop_conn.retr(1)

To gracefully close the connection, we send a QUIT command:

>>> pop_conn.quit()
b'+OK Logging out.'

For a specification of POP3, see RFC 1939.

MIME: Multipurpose Internet Mail Extension

Mail client apps often have the feature to view the message in it’s raw form. One would notice that email messages consist of headers and payload, not unlike HTTP requests and and responses.

A very simple MIME message would be something like:

Content-Type: text/plain
MIME-Version: 1.0
To: <developer@evilsoft.com>
From: <developer@fb.com>
Subject: Sample MIME Message

This is a simple MIME Message!

Headers being used here are pretty much self-explanatory. This kind of message is a single part one - there is a single type of payload of given content type. For plain text messages message contents are transferred in their original form. If the payload was of some binary format (e.g. PDF document or image file) it would be represented in Base64 form. Some of the popular MIME content types are:

text/plain - just a plain text
text/html - HTML document
image/png - PNG image
application/pdf - PDF document

But nowadays many, if not most, email messages are more complex than this. MIME messages can also be multipart - consisting of several segments.

RFC 1341 provides the following example of two-part MIME message:

From: Nathaniel Borenstein <nsb@bellcore.com> 
To:  Ned Freed <ned@innosoft.com> 
Subject: Sample message 
MIME-Version: 1.0 
Content-type: multipart/mixed; boundary="simple 
boundary" 

This is the preamble.  It is to be ignored, though it 
is a handy place for mail composers to include an 
explanatory note to non-MIME compliant readers. 
--simple boundary 

This is implicitly typed plain ASCII text. 
It does NOT end with a linebreak. 
--simple boundary 
Content-type: text/plain; charset=us-ascii 

This is explicitly typed plain ASCII text. 
It DOES end with a linebreak. 

--simple boundary-- 
This is the epilogue.  It is also to be ignored.

Multipart messages can also be mixed and alternative. Mixed messages (content type multipart/mixed) are multipart messages with parts of different content types that are independent and meant to be displayed serially (one after another). Alternative messages (content type multipart/alternative contain at least two parts that are equivalent representation of the same information (e.g. the same message could be in plain text and HTML). MIME messages are powerful enough to form a tree structure with message as a root node and parts as descendant nodes. That way, parts of message could have child parts with files or previous messages as attachments.

Various modules in Python email package deal with parsing and generating the MIME messages.

Wget tips and tricks

Fri, 11 Nov 2022 00:00:00 +0000

So, what is wget? Wget is a prominent command line tool to download stuff from the web. It has a fairly extensive feature set that includes recursive downloading, proxy support, site mirroring, cookie handling and so on. Let us go through some use cases of wget.

To download a single file/page using wget, just pass the corresponding URL into argv[1]:

$ wget http://www.textfiles.com/100/crossbow
--2022-11-11 18:24:33--  http://www.textfiles.com/100/crossbow
Resolving www.textfiles.com (www.textfiles.com)... 208.86.224.90
Connecting to www.textfiles.com (www.textfiles.com)|208.86.224.90|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 29200 (29K)
Saving to: ‘crossbow’

crossbow                                           100%[================================================================================================================>]  28,52K   114KB/s    in 0,3s    

2022-11-11 18:24:33 (114 KB/s) - ‘crossbow’ saved [29200/29200]

This shows us the HTTP status code, server IP address, file size and even visualises the progress on ASCII-art progress bar. This is default level of output verbosity. But what if we wanted to see more details? Perhaps something is off and we want to troubleshoot it? We can enable debug output by using --debug:

$ wget --debug http://www.textfiles.com/100/crossbow
DEBUG output created by Wget 1.21.3 on darwin21.3.0.

Reading HSTS entries from /Users/rl/.wget-hsts
URI encoding = ‘UTF-8’
Converted file name 'crossbow' (UTF-8) -> 'crossbow' (UTF-8)
--2022-11-11 18:26:48--  http://www.textfiles.com/100/crossbow
Resolving www.textfiles.com (www.textfiles.com)... 208.86.224.90
Caching www.textfiles.com => 208.86.224.90
Connecting to www.textfiles.com (www.textfiles.com)|208.86.224.90|:80... connected.
Created socket 5.
Releasing 0x0000600000a3aae0 (new refcount 1).

---request begin---
GET /100/crossbow HTTP/1.1
Host: www.textfiles.com
User-Agent: Wget/1.21.3
Accept: */*
Accept-Encoding: identity
Connection: Keep-Alive

---request end---
HTTP request sent, awaiting response... 
---response begin---
HTTP/1.1 200 OK
Date: Fri, 11 Nov 2022 10:26:49 GMT
Server: Apache/2.4.54 (FreeBSD) OpenSSL/1.1.1o-freebsd
Last-Modified: Sun, 01 Aug 1999 17:20:50 GMT
ETag: "7210-35109efcee080"
Accept-Ranges: bytes
Content-Length: 29200
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive

---response end---
200 OK
Registered socket 5 for persistent reuse.
Length: 29200 (29K)
Saving to: ‘crossbow.1’

crossbow.1                                         100%[================================================================================================================>]  28,52K   112KB/s    in 0,3s    

2022-11-11 18:26:49 (112 KB/s) - ‘crossbow.1’ saved [29200/29200]

We got the HTTP message headers and even some socket lifecycle logs.

If we wanted to make output less verbose than it usually is, we could use --no-verbose:

$ wget --no-verbose http://www.textfiles.com/100/crossbow
2022-11-11 18:30:32 URL:http://www.textfiles.com/100/crossbow [29200/29200] -> "crossbow.2" [1]

To make the wget print no output, run it with --quiet.

If you want to download something from legacy FTP server you can do it with wget. Running wget with directory URL fetches the file list for that directory and converts it to HTML (don’t forget slash character at the end):

$ wget ftp://ftp.cs.brown.edu/pub/
--2022-11-11 18:37:22--  ftp://ftp.cs.brown.edu/pub/
           => ‘.listing’
Resolving ftp.cs.brown.edu (ftp.cs.brown.edu)... 128.148.32.111
Connecting to ftp.cs.brown.edu (ftp.cs.brown.edu)|128.148.32.111|:21... connected.
Logging in as anonymous ... Logged in!
==> SYST ... done.    ==> PWD ... done.
==> TYPE I ... done.  ==> CWD (1) /pub ... done.
==> PASV ... done.    ==> LIST ... done.

.listing                                               [ <=>                                                                                                             ]   6,16K  --.-KB/s    in 0s      

2022-11-11 18:37:24 (55,2 MB/s) - ‘.listing’ saved [6312]

Removed ‘.listing’.
Wrote HTML-ized index to ‘index.html’ [9267].

Screenshot 1

Once we have chosen a file to download, we can do it the same way as with HTTP protocol:

$ wget ftp://ftp.cs.brown.edu/pub/README 
--2022-11-11 18:39:53--  ftp://ftp.cs.brown.edu/pub/README
           => ‘README’
Resolving ftp.cs.brown.edu (ftp.cs.brown.edu)... 128.148.32.111
Connecting to ftp.cs.brown.edu (ftp.cs.brown.edu)|128.148.32.111|:21... connected.
Logging in as anonymous ... Logged in!
==> SYST ... done.    ==> PWD ... done.
==> TYPE I ... done.  ==> CWD (1) /pub ... done.
==> SIZE README ... 7998
==> PASV ... done.    ==> RETR README ... done.
Length: 7998 (7,8K) (unauthoritative)

README                                             100%[================================================================================================================>]   7,81K  --.-KB/s    in 0,006s  

2022-11-11 18:39:56 (1,25 MB/s) - ‘README’ saved [7998]

Now, what if we wanted wget to follow links and traverse pages recursively to download entire website? This is called website mirroring and wget has multiple CLI options to support this use case:

--adjust-extension fixes the extension of downloaded HTML pages.
--recursive, -r enables recursive crawling (default depth is 5).
--level, -l specifies crawl depth.
--convert-links, -k converts links in HTML pages so that they are suitable for local viewing.
--domains, -D specifies a comma-separated list of domains to set boundaries when crawling.
--mirror enables mirroring mode.

For example, we could run:

$ wget --mirror --domains quotes.toscrape.com,fonts.googleapis.com,fonts.gstatic.com --convert-links  https://quotes.toscrape.com

This would download a static website for offline browsing. Note, however, that this is pretty much limited to static websites that are based on simple HTML. This technique will not be able to replicate the API calls and other stuff that JS-heavy sites are doing in the browser.

Screenshot 2

Now, what do we do if there’s a login wall or something similar standing in our way? We may need to submit a login form or press a button to get a cookie that would allow us to download a file. Fortunately, we can use wget with --post-data to submit form data and --save-cookies to save cookies into file. We also need to use --keep-session-cookies so that cookies without expiration date would be saved as well.

The command to reproduce VMWare Customer Connect login action would be the following:

$ wget --keep-session-cookies --save-cookies cookies.txt --post-data "username=[REDACTED]&password=[REDACTED]" "https://auth.vmware.com/oam/server/auth_cred_submit?Auth-AppID=WMVMWR"

Now we have some cookies saved into text file. We can use them with --load-cookies to download some software we have from VMWare:

$ wget --load-cookies cookies.txt https://download3.vmware.com/software/FUS-1224/VMware-Fusion-12.2.4-20071091_x86.dmg

The last use case we want to look into is user wget as a crawler that saves a list of pages it goes through as it traverses the site. To this purpose we can use --spider and --delete-after with some of the CLI options we utilised when we did the mirroring:

$ wget --no-verbose --recursive --domains quotes.toscrape.com,fonts.googleapis.com,fonts.gstatic.com --spider --delete-after https://quotes.toscrape.com 2>&1 | tee wget_urls.txt 
2022-11-11 20:25:07 URL:https://quotes.toscrape.com/ [11053/11053] -> "quotes.toscrape.com/index.html.tmp.tmp" [1]
https://quotes.toscrape.com/robots.txt:
2022-11-11 20:25:07 ERROR 404: NOT FOUND.
2022-11-11 20:25:10 URL:https://quotes.toscrape.com/static/bootstrap.min.css [125934/125934] -> "quotes.toscrape.com/static/bootstrap.min.css.tmp.tmp" [1]
2022-11-11 20:25:11 URL:https://quotes.toscrape.com/static/main.css [1370/1370] -> "quotes.toscrape.com/static/main.css.tmp.tmp" [1]
2022-11-11 20:25:11 URL:https://quotes.toscrape.com/login [1869/1869] -> "quotes.toscrape.com/login.tmp.tmp" [1]
2022-11-11 20:25:12 URL:http://quotes.toscrape.com/author/Albert-Einstein/ [5324/5324] -> "quotes.toscrape.com/author/Albert-Einstein.tmp.tmp" [1]
http://quotes.toscrape.com/robots.txt:
2022-11-11 20:25:13 ERROR 404: NOT FOUND.
2022-11-11 20:25:14 URL:https://quotes.toscrape.com/tag/change/page/1/ [3851/3851] -> "quotes.toscrape.com/tag/change/page/1/index.html.tmp.tmp" [1]
2022-11-11 20:25:15 URL:https://quotes.toscrape.com/tag/deep-thoughts/page/1/ [3865/3865] -> "quotes.toscrape.com/tag/deep-thoughts/page/1/index.html.tmp.tmp" [1]
2022-11-11 20:25:16 URL:https://quotes.toscrape.com/tag/thinking/page/1/ [4687/4687] -> "quotes.toscrape.com/tag/thinking/page/1/index.html.tmp.tmp" [1]
2022-11-11 20:25:16 URL:https://quotes.toscrape.com/tag/world/page/1/ [3849/3849] -> "quotes.toscrape.com/tag/world/page/1/index.html.tmp.tmp" [1]
2022-11-11 20:25:18 URL:http://quotes.toscrape.com/author/J-K-Rowling/ [5343/5343] -> "quotes.toscrape.com/author/J-K-Rowling.tmp.tmp" [1]
2022-11-11 20:25:19 URL:https://quotes.toscrape.com/tag/abilities/page/1/ [3638/3638] -> "quotes.toscrape.com/tag/abilities/page/1/index.html.tmp.tmp" [1]
2022-11-11 20:25:19 URL:https://quotes.toscrape.com/tag/choices/page/1/ [3634/3634] -> "quotes.toscrape.com/tag/choices/page/1/index.html.tmp.tmp" [1]
2022-11-11 20:25:21 URL:https://quotes.toscrape.com/tag/inspirational/page/1/ [12815/12815] -> "quotes.toscrape.com/tag/inspirational/page/1/index.html.tmp.tmp" [1]
2022-11-11 20:25:22 URL:https://quotes.toscrape.com/tag/life/page/1/ [12648/12648] -> "quotes.toscrape.com/tag/life/page/1/index.html.tmp.tmp" [1]
2022-11-11 20:25:22 URL:https://quotes.toscrape.com/tag/live/page/1/ [3942/3942] -> "quotes.toscrape.com/tag/live/page/1/index.html.tmp.tmp" [1]
2022-11-11 20:25:23 URL:https://quotes.toscrape.com/tag/miracle/page/1/ [3948/3948] -> "quotes.toscrape.com/tag/miracle/page/1/index.html.tmp.tmp" [1]
2022-11-11 20:25:24 URL:https://quotes.toscrape.com/tag/miracles/page/1/ [3950/3950] -> "quotes.toscrape.com/tag/miracles/page/1/index.html.tmp.tmp" [1]
...

The output needs a little cleanup, but essentially we are getting a list of pages on the site. This can save your day if for some reason you cannot install hakrawler or similar tool when doing bounty hunting, but really need some site crawled.

Axiom: just-in-time dynamic infra for offensive security operations

Mon, 07 Nov 2022 00:00:00 +0000

When doing recon part of penetration testing or bug bounty hunting engagements one may want to run various tools (such as port scanners, crawlers, vulnerability scanners, headless browsers and so on) in a VPS environment that will no longer be needed when the task at hand is complete. For large-scale scanning it is highly desirable to spread the workload across many servers. To address these need, axiom was developed. Axiom is a dynamic infrastructure framework designed for quick setup and teardown of reproducible infrastructure.

0xtavian, one of the founding developers of this project, was doing security work and has found that there’s a great variety of security tools, but no established way to use them in unified way to easily set up an VPS for hacking. Furthermore, security centric Linux distros such as Kali provided a variety of tools out of the box, but lacked tooling for truly scalable bounty hunting and pentesting. These two problems motivated 0xtavian to create Axiom with pry0cc, another developer in security space.

Let us install Axiom into VPS instance running one of the operating systems that support the easy install:

Debian Linux
Ubuntu Linux
Kali Linux
Windows (with Ubuntu on WSL)
macOS

I am recommending doing this in a VPS environment because Axiom installs some packages on your local system through APT or Homebrew that you may not need otherwise. Furthermore, you may want to run Axiom scans in tmux session on the server for longer scans that may take overnight or multiple days.

The easy install command is the following:

bash <(curl -s https://raw.githubusercontent.com/pry0cc/axiom/master/interact/axiom-configure)

This will download Axiom source code, install dependencies, establish API connection to cloud provider (Digital Ocean, Linode, Azure, IBM Cloud or AWS) - you will be asked to provide API credentials for your cloud providers, as well as some other details - and use Packer to create VPS image with all the tooling to be used for scanning. For the purpose of trying it out choose the default provisioner.

Once installation is complete, let us start small by setting up a single VPS with axiom-init command (this should take a few minutes):

$ axiom-init first
Initializing 'first' at 'lon1' with image 'axiom-default-1666592377'
INITIALIZING IN 5 SECONDS, CTRL+C to quit... 
Initialized instance 'first' at '[REDACTED]'!
To connect, run 'axiom-ssh first' or 'axiom-connect'

Now let us run a single command through axiom-exec on the worker server:

$ axiom-exec 'dig google.com'
=====================================================
Interlace v1.9.6	by Michael Skelton (@codingo_)
                  	& Sajeeb Lohani (@sml555_)
=====================================================
  0%|                                                                                                                                                                                 | 0/1 [00:00<?, ?it/s][11:19:50] [THREAD] [ssh -F /root/.axiom/tmp/exec/axiom-exec+1667819982/sshconfig -o StrictHostKeyChecking=no -o PasswordAuthentication=no first 'tmux new -d -s axiom-exec+1667819982 && tmux send-keys -t axiom-exec+1667819982 "bash -i /tmp/exec/axiom-exec+1667819982/command  > >(tee -a /tmp/exec/axiom-exec+1667819982/stdout.log) 2> >(tee -a /tmp/exec/axiom-exec+1667819982/stderr.log >&2) ; touch /tmp/exec/axiom-exec+1667819982/first" ENTER ' "&& tmux send-keys -t axiom-exec+1667819982 exit ENTER"] Added to Queue 
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 16.72it/s]

; <<>> DiG 9.16.1-Ubuntu <<>> google.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 7177
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;google.com.			IN	A

;; ANSWER SECTION:
google.com.		31	IN	A	142.250.187.206

;; Query time: 0 msec
;; SERVER: 127.0.0.53#53(127.0.0.53)
;; WHEN: Mon Nov 07 11:19:50 UTC 2022
;; MSG SIZE  rcvd: 55

Axiom ships with bunch of recipes (modules) that integrate various scanning tools into the system. Let us try running Nmap on the worker server. First, we need to create an input file with target hostname(s):

$ echo "scanme.nmap.org" >> target.txt

Then we can run axiom-scan to run a scan with Nmap module and our text file as input:

$ axiom-scan target.txt -m nmap -o nmap_out.txt -oG nmap_out.txt
sorting and greping unique for: 'nmap_out.txt'
              _
  ____ __  __(_)___  ____ ___        ______________ _____
 / __ `/ |/_/ / __ \/ __ `__ \______/ ___/ ___/ __ `/ __ \
/ /_/ />  </ / /_/ / / / / / /_____(__  ) /__/ /_/ / / / /
\__,_/_/|_/_/\____/_/ /_/ /_/     /____/\___/\__,_/_/ /_/

                                    @pry0cc
                                 & @0xtavian

creating scan working directory at : /home/op/scan/nmap+166782070320617/
module: [ nmap ] | module args: [  ] | input: [ 1 lines ] |
instances:  1  [ first ] |
command: [ sudo nmap -iL input -oG output ] | ext: [ txt ] | threads: [ null ]
spliting and distributing input file...
[ OK ]
Starting Nmap 7.92 ( https://nmap.org ) at 2022-11-07 11:31 UTC
Stats: 0:00:00 elapsed; 0 hosts completed (0 up), 1 undergoing Ping Scan
Ping Scan Timing: About 100.00% done; ETC: 11:31 (0:00:00 remaining)
Nmap scan report for scanme.nmap.org (45.33.32.156)
Host is up (0.15s latency).
Other addresses for scanme.nmap.org (not scanned): 2600:3c01::f03c:91ff:fe18:bb2f
Not shown: 996 closed tcp ports (reset)
PORT      STATE SERVICE
22/tcp    open  ssh
80/tcp    open  http
9929/tcp  open  nping-echo
31337/tcp open  Elite

Nmap done: 1 IP address (1 host up) scanned in 2.53 seconds
first scan finished
Mode set to txt.. Sorting unique.
Appending axiom-scan runtime statistics to : /root/.axiom/stats.log
module: [ nmap ] | module args: [  ] | instances: [ 1 ] | targets: [ 1 targets ] | results: [ 4 results ] |
runtime: [ 00h:00m:41s ] | date: [ Mon Nov  7 11:31:43 UTC 2022 ] | id: [ nmap+166782070320617 ] |
output: [ /root/nmap_out.txt ] | log: [ /root/.axiom/logs/nmap+166782070320617 ] | remote: [ /home/op/scan/nmap+166782070320617 ]  |
command: [ sudo nmap -iL input -oG output ] | ext: [ txt ] | threads: [ null ]

We used -m to select an Axiom module and -o for output file. After that, we can pass the CLI arguments to the tool that Axiom would run - we used -oG for selecting a greppable output.

To list currently running instances, we can run axiom-ls. To make a backup of an instance, we can use axiom-backup with the name of an instance (to be restored using axiom-restore).

But for now, we want to simply remove an instance we have created. We are doing this with axiom-rm:

$ axiom-rm first
Deleting 'first'...
Warning: Are you sure you want to delete this Droplet? (y/N) ? y

Now we want to go bigger. To fully benefit from Axiom we want to create a fleet - a set of worker servers that will perform the workloads in a load-balanced way. Let us create a fleet of 8 servers.

$ axiom-fleet -i 8

We will try running some scans against Firing Range - a deliberately insecure website developed for testing web security scanners. Let us put the corresponding URL to target.txt file:

$ echo "https://public-firing-range.appspot.com/" > target.txt

To crawl the site and save all the page URLs into a textfile, let us run Hakrawler through Axiom:

$ axiom-scan target.txt -m hakrawler -o crawl.txt

That gave us a list of 287 URLs:

$ wc -l crawl.txt 
287 crawl.txt

This new list can be used as input file for the next scan that will entail running Nuclei vulnerability scanner:

$ axiom-scan crawl.txt -m nuclei -o nuclei.txt

Axiom now splits the input file into even parts across all worker server so that workload is distributed evenly, runs Nuclei on them via SSH, waits for it to finish, downloads the results, deduplicates them and saves into a textfile. Large scans can take a while even when done across several servers, thus I recommend doing them from a tmux session.

There will be quite a bit of verbose output in the terminal that mostly matters for seeing progress and troubleshooting, thus is not reproduced here. What matters is stuff in the output file. Not everything here is something that warrants further attention, but some of the entries will be marked as medium or high in severity and in real engagement would need to be investigated further.

To remove the fleet, we can use axiom-rm with server name prefix:

$ axiom-rm "kalam*" -f

To daisy-chain several scans into larger workload one can trivially write a Bash script consisting of Axiom commands in the order you would run them on the shell.

Scraping legacy ASP.Net sites with Scrapy: a real world example

Wed, 12 Oct 2022 00:00:00 +0000

Legacy ASP.Net sites are some of the most difficult ones to deal with for web scraper developers due to their archaic state management mechanism that entails hidden form submissions via POST requests. This stuff is not easy to reproduce programmatically. In this post we will discuss a real world example of scraping a site based on some old version of ASP.Net technology.

The site in question is publicaccess.claytoncounty.gov - a real estate record website for Clayton County, Georgia. Our objective is to scrape all the real estate sales records from this site (fun fact: there’s plenty of juicy data various branches of US government publish on official sites that can be scraped and monetised). We will be using Scrapy framework. It is assumed that the reader is already familiar with this framework, thus some of the basic technical details will be skipped. Data we want to scrape will correspond to the following subclass of scrapy.Item:

class SalesItem(scrapy.Item):
    # define the fields for your item here like:
    # name = scrapy.Field()
    state = scrapy.Field()
    property_zip5 = scrapy.Field()
    property_street_address = scrapy.Field()
    property_city = scrapy.Field()
    property_county = scrapy.Field()
    property_id = scrapy.Field()
    sale_datetime = scrapy.Field()
    property_type = scrapy.Field()
    sale_price = scrapy.Field()
    seller_1_name = scrapy.Field()
    buyer_1_name = scrapy.Field()
    building_num_units = scrapy.Field()
    building_year_built = scrapy.Field()
    source_url = scrapy.Field()
    book = scrapy.Field()
    page = scrapy.Field()
    transfer_deed_type = scrapy.Field()
    property_township = scrapy.Field()
    property_lat = scrapy.Field()
    property_lon = scrapy.Field()
    sale_id = scrapy.Field()
    deed_date = scrapy.Field()
    building_num_stories = scrapy.Field()
    building_num_beds = scrapy.Field()
    building_num_baths = scrapy.Field()
    building_area_sqft = scrapy.Field()
    building_assessed_value = scrapy.Field()
    building_assessed_date = scrapy.Field()
    land_area_acres = scrapy.Field()
    land_area_sqft = scrapy.Field()
    land_assessed_value = scrapy.Field()
    seller_2_name = scrapy.Field()
    buyer_2_name = scrapy.Field()
    land_assessed_date = scrapy.Field()
    seller_1_state = scrapy.Field()
    seller_2_state = scrapy.Field()
    buyer_1_state = scrapy.Field()
    buyer_2_state = scrapy.Field()
    total_assessed_value = scrapy.Field()
    total_appraised_value = scrapy.Field()
    land_appraised_value = scrapy.Field()
    building_appraised_value = scrapy.Field()
    land_type = scrapy.Field()

We don’t need to fill all the fields here, just the ones that are available from the source website.

When we endeavor to scrape a non-trivial site we first need to do some exploration to learn about the site so that we could strategise and plan development efforts.

If we load the site without any cookies being present in the browser we will be redirected to disclaimer page that the county has made to cover its ass legally. This will also give us the very first cookie with ASP.Net session ID.

Screenshot 1

Inspecting the DOM tree reveals that pressing “Agree” button on this page would cause a hidden form to be submitted. This form contains __VIEWSTATE, __VIEWSTATEGENERATOR and __EVENTVALIDATION fields with pre-filled valued that deal with client session state management. We will have to submit these values as they are without any changes when we are going to reproduce the button press in the Scrapy spider. The same applies to our further interactions with the site.

Screenshot 2

Pressing the “Agree” button submits the hidden form via POST requests and redirects us to a page with some search functionality. At this point we want to get a complete list of all the real estate property records that are available, but merely pressing the search button on this form tells us to fill in at least one search criteria.

Screenshot 3

In the page header there’s links to other search pages. “Real Property Search” is the default one and “Advanced Search” provides us with more options/criteria to search records by. We will be using this one. Others are not that relevant to us. “Map Search” didn’t even work for me and “Sales Search” is similar to “Advanced Search”, but with less stuff to search by. So we go to “Advanced Search” and try to get all records from 1900 January 1 to 2022 December 31 (it does not allow us to search earlier in time than 1900). Note, however, that the default form is not completely useless to us - it can be used with ID of specific parcel to check the scraped data manually since we will not be able to fill unique URL into source_url field.

Screenshot 4

This did give us some search results, but not all of them as the site caps the number of results to 10 000. This is a common problem when doing web scraping and it can be solved by doing search what I call search space sharding. Instead of searching for entire recorded history, we will search in smaller increments. A little experimentation reveals that searching for one month at a time allows each search result list to fit within the limit of 10 000 results.

Screenshot 5

Let us go a bit deeper now. What happens when Search button is pressed? Well, we have another form submission with bunch of crazy fields in the payload. Some of values for this form are available in the original HTML page. Some are not and are filled by client side JS code instead (based on search criteria the user entered). Some can be safely ignored. A key challenge in scraping ASP.Net sites is reproducing this kind of requests accurately enough. We will be dealing with this later.

Screenshot 6

What if we click one of the rows in the results table? There is another POST request to the same endpoint, but somewhat different form data being submitted. For example, hdLink contains a relative link to a page we get redirected to. We don’t get a unique URL to parcel page as accessing this exact page is predicated on client state earlier in the session and the form data in the request that happens when result row is clicked.

Screenshot 7

So where does it get the value for hdLink from? If we go to search results page and inspect the row, we will find that each row (tr element) has onclick property with JS snippet like the following:

javascript:selectSearchRow('../Datalets/Datalet.aspx?sIndex=3&idx=1')

Screenshot 8

We can scrape this from HTML document and use it in our code. There will be more finer parts of the POST request that we need to get right, but let us leave them for later.

When we go see the result we find that data about parcel is broken down across several pages. The first page we get is “Tax Commisioner Summary” page that contains some information on property address, most common owner, property class and so on. There are links on the side to further pages (no hidden forms this time, but we’re not getting unique URLs either).

Screenshot 9

We will be scraping these pages:

“Tax Commissioner Summary” for property_id, property_street_address and property_type fields.
“Residential” - building_year_built, building_num_beds, building_num_baths.
“Value History” - assessed/appraised monetary values (land, buildings and total). We will need to make sure that the year of appraisal/assessment matches that of a sale for each row we output.
“Land” - land_area_acres, land_area_sqft.
“Sales” - everything about parcel sale: sale_datetime, sale_price, seller_1_name, buyer_1_name, seller_2_name, buyer_2_name, book, page and transfer_deed_type.

There’s one sale per page, but sale pages form a linked list through the little navigation links that we can use.

Screenshot 10

The same applies to parcels - there’s a way to navigate between adjacent search results (based on form submission this time - we will need to work out details here).

Screenshot 11

This lets us have a linear flow between pages without the inconvenience of backtracking.

Based on what we learned so far the scraping strategy will be as follows:

Shard search space by searching for one month worth of results per search query.
- For each search, go to the first result (reproduce form submission).
  - Scrape data from “Tax Commisioner Summary”, “Residential”, “Value History”, “Land” pages. Use meta dictionary to pass information between requests so that stuff is being scraped incrementally.
  - Scrape sales data by traversing linked list of sales records. Create items here by filling stuff from sales record and stuff from earlier. Make sure that appraised/ assessed values match the sales year. Refrain from including them if it’s not possible.
- Go to next parcel so that list of parcels is traversed to the end.

Due to the nature of legacy ASP.Net technology we cannot benefit much from built-in concurrency of Scrapy at session level - we risk messing up the session if we don’t load pages one at a time. However we can have multiple sessions at the same time - we will be writing some extra code to make the scraping faster and more reliable.

Let us go through the Scrapy spider code now, one step at a time.

We start scraping by loading the disclaimer page (this will give us the ASP.Net session cookie) and reproducing the action of user pressing the “Agree” button:

import scrapy

from scrapy.http import FormRequest

class ClaytonSpider(scrapy.Spider):
    name = "clayton"
    allowed_domains = ["publicaccess.claytoncountyga.gov"]
    start_urls = [
        "https://publicaccess.claytoncountyga.gov/Search/Disclaimer.aspx?FromUrl=../search/advancedsearch.aspx?mode=advanced",
        "https://publicaccess.claytoncountyga.gov/search/advancedsearch.aspx?mode=advanced",
    ]
    start_year = 1939
    state = "GA"
    county = "CLAYTON"
    shards = []

    def start_requests(self):
        yield scrapy.Request(self.start_urls[0], callback=self.parse_disclaimer_page)

    def parse_disclaimer_page(self, response):
        yield FormRequest.from_response(
            response,
            formxpath='//form[@name="Form1"]',
            clickdata={"name": "btAgree"},
            callback=self.parse_form_page,
        )

In this case we are able to use from_response utility method of FormRequest class.

We have skipped this part for now, but shards list will be filled with year-month tuples to be used in search step.

Callback to request that submits the agreement to legal terms is parse_form_page():

    def parse_form_page(self, response):
        if len(self.shards) == 0:
            yield None
            return

        year, month = self.shards.pop()
        logging.info("Year: {}, month: {}".format(year, month))

        days_in_month = calendar.monthrange(year, month)[1]

        start_date = date(year=year, month=month, day=1)
        end_date = date(year=year, month=month, day=days_in_month)

        yield self.make_search_form_request(response, start_date, end_date)

Now we have a form we can fill, thus we remove one search shard from the list and use it to fill the search form. Since we will be doing this a lot, we have another method for this purpose:

    def make_search_form_request(self, response, start_date, end_date):
        start_date_str = self.stringify_date(start_date)
        end_date_str = self.stringify_date(end_date)

        form_data = self.extract_form(response, '//form[@name="frmMain"]')

        form_data["hdCriteria"] = "salesdate|{}~{}".format(start_date_str, end_date_str)
        form_data["ctl01$cal1"] = start_date.isoformat()
        form_data["ctl01$cal1$dateInput"] = start_date_str
        form_data["txtCrit"] = start_date_str
        form_data["ctl01$cal2"] = end_date.isoformat()
        form_data["ctl01$cal2$dateInput"] = end_date_str
        form_data["txtCrit2"] = end_date_str
        form_data["PageNum"] = "1"
        form_data["PageSize"] = "1"
        form_data["hdCriteriaTypes"] = "N|N|C|C|C|N|C|C|N|D|N|N|C|C|C|N|N"
        form_data["sCriteria"] = "0"

        logging.debug(form_data)

        return FormRequest(
            response.url,
            formdata=form_data,
            callback=self.parse_search_result_list,
            dont_filter=True,
        )

You can see that it calls a helper method to extract form fields already present in the page, but also fills bunch of fields explicitly. This is significant. How do I know that these form fields have to be filled in this exact way? Well, through experimentation, persistance and plain old paying the fucking attention. It took me quite a bit of effort to work these exact details out, but there was nothing magical or secret in doing that. There was multiple takes of painstakingly comparing what logging.debug() logs out with what I see in the Chrome dev tools until I got it working correctly.

Next, we get to parse the HTML page with result list:

    def parse_search_result_list(self, response):
        first_row = response.xpath('//tr[@class="SearchResults"][1]')
        if len(first_row) != 1:
            yield scrapy.Request(
                self.start_urls[-1], callback=self.parse_form_page, dont_filter=True
            )
            return

        logging.debug(first_row.xpath(".//text()").getall())

        onclick = first_row.attrib.get("onclick")
        rel_url = onclick.replace("javascript:selectSearchRow('", "").replace("')", "")

        form_data = self.extract_form(response, '//form[@name="frmMain"]')
        form_data["hdLink"] = rel_url
        form_data["hdAction"] = "Link"
        form_data["hdSelectAllChecked"] = "false"
        form_data["sCriteria"] = "0"
        logging.debug(form_data)

        action = response.xpath('//form[@name="frmMain"]/@action').get()
        form_url = urljoin(response.url, action)

        yield FormRequest(
            form_url,
            formdata=form_data,
            callback=self.parse_property_main_page,
            dont_filter=True,
        )

We try to extract the first row of the results table. If it’s not there we take it that there’s no records for the given month, thus we move on to the next shard in the list. If it is there, we extract onclick property, do some basic string manipulation to extract the relative link from that, fill the hidden form and submit it. This will give us a parcel summary page on the next callback.

The next callback is parse_property_main_page():

    def parse_property_main_page(self, response):
        parid = response.xpath('//input[@id="hdXPin"]/@value').get()
        property_street_address = (
            response.xpath('//td[@class="DataletHeaderBottom"][last()]/text()')
            .get("")
            .strip()
        )
        property_street_address = " ".join(
            property_street_address.split()
        )  # https://stackoverflow.com/a/1546251

        item = SalesItem()
        item["state"] = self.state
        item["property_id"] = parid
        item["property_street_address"] = property_street_address
        item["property_county"] = self.county
        item["property_type"] = response.xpath(
            '//tr[./td[contains(text(),"Property Class")]]/td[@class="DataletData"]/text()'
        ).get()

        residential_link = response.xpath(
            '//a[./span[text()="Residential"]]/@href'
        ).get()
        yield response.follow(
            residential_link,
            meta={"item": item},
            callback=self.parse_property_residential_page,
            dont_filter=True,
        )

Here we do some XPath queries to extract some fields into newly created item object. We also extract link to “Residential” page and go there. Since we’re not done filling the item, we pass it along the request by using the meta dictionary.

At next callback we scrape the “Residential” page:

    def parse_property_residential_page(self, response):
        item = response.meta.get("item")

        parid = response.xpath('//input[@id="hdXPin"]/@value').get()
        assert parid == item["property_id"]

        item["building_year_built"] = response.xpath(
            '//tr[./td[text()="Year Built"]]/td[@class="DataletData"]/text()'
        ).get()
        item["building_num_beds"] = response.xpath(
            '//tr[./td[text()="Bedrooms"]]/td[@class="DataletData"]/text()'
        ).get()
        item["building_num_baths"] = response.xpath(
            '//tr[./td[text()="Full Baths"]]/td[@class="DataletData"]/text()'
        ).get()

        value_history_link = response.xpath(
            '//a[./span[text()="Value History"]]/@href'
        ).get()
        yield response.follow(
            value_history_link,
            meta={"item": item},
            callback=self.parse_property_value_history_page,
            dont_filter=True,
        )

We get the item object from response.meta, fill in some more fields and pass it along the next request that will fetch us “Value History” page that we parse at next callback:

    def parse_property_value_history_page(self, response):
        item = response.meta.get("item")
        appr_rows = response.meta.get("appr_rows")
        as_rows = response.meta.get("as_rows")

        appr_rows = dict()
        appr_header = None

        appr_values_table = response.xpath('//table[@id="Appraised Values"]')
        if appr_values_table is not None:
            for row in appr_values_table.xpath("./tr"):
                row = row.xpath("./td/text()").getall()
                if appr_header is None:
                    appr_header = row
                    continue

                if len(row) == 0:
                    continue

                year = row[0]
                try:
                    year = int(year)
                except:
                    continue
                row = dict(zip(appr_header, row))
                appr_rows[year] = row

        logging.debug(appr_rows)

        as_rows = dict()
        as_header = None

        as_values_table = response.xpath('//table[@id="Assessed Values"]')
        if as_values_table is not None:
            for row in as_values_table.xpath("./tr"):
                row = row.xpath("./td/text()").getall()
                if as_header is None:
                    as_header = row
                    continue

                if len(row) == 0:
                    continue

                year = row[0]
                try:
                    year = int(year)
                except:
                    continue
                row = dict(zip(as_header, row))
                as_rows[year] = row

        logging.debug(as_rows)

        meta_dict = {"item": item, "appr_rows": appr_rows, "as_rows": as_rows}

        land_link = response.xpath('//a[./span[text()="Land"]]/@href').get()
        yield response.follow(
            land_link,
            meta=meta_dict,
            callback=self.parse_property_land_page,
            dont_filter=True,
        )

Now we have two tables. One for assessed values and one for appraised values (these are different ways to evaluate real estate prices). In each row there will be year and numbers for land, building and overall value. For some parcels these tables will be missing, which means we have to gracefully skip this page without crashing the scraping job. So what we do is look for tables via XPath queries, scrape their rows and store the results in dictionaries indexed by years. Since this data will be needed later we pass it along the request through meta dictionary.

Screenshot 12

At next callback, “Land” page is scraped:

    def parse_property_land_page(self, response):
        item = response.meta.get("item")
        appr_rows = response.meta.get("appr_rows")
        as_rows = response.meta.get("as_rows")

        item["land_area_acres"] = (
            response.xpath('//tr[./td[text()="Acres"]]/td[@class="DataletData"]/text()')
            .get("")
            .replace("\xa0", "")
        )
        if item["land_area_acres"].startswith("."):
            item["land_area_acres"] = 0
        elif item["land_area_acres"] != "":
            item["land_area_acres"] = round(float(item["land_area_acres"]))
        item["land_area_sqft"] = (
            response.xpath(
                '//tr[./td[text()="Square Feet"]]/td[@class="DataletData"]/text()'
            )
            .get("")
            .replace("\xa0", "")
            .replace(",", "")
        )
        item["land_type"] = response.xpath(
            '//tr[./td[text()="Land Type"]]/td[@class="DataletData"]/text()'
        ).get()

        meta_dict = {"item": item, "appr_rows": appr_rows, "as_rows": as_rows}

        sales_link = response.xpath('//a[./span[text()="Sales"]]/@href').get()
        yield response.follow(
            sales_link,
            meta=meta_dict,
            callback=self.parse_property_sales_page,
            dont_filter=True,
        )

We extract some more fields into the item object and go to the “Sales” page, like we did in previous steps. Note that meta dictionary has to be recreated like it is being done here. One cannot take a reference to meta from response object, change something it a bit and pass it into new request object, as Scrapy framework also sets some key-value pairs in this dictionary for internal use. Not recreating the meta dictionary will mess up the Scrapy internals.

So finally we get to scrape the sales data at in parse_property_sales_page():

    def parse_property_sales_page(self, response):
        item = response.meta.get("item")
        appr_rows = response.meta.get("appr_rows")
        as_rows = response.meta.get("as_rows")

        sale_date_str = response.xpath(
            '//tr[./td[text()="Sale Date"]]/td[@class="DataletData"]/text()'
        ).get()
        if sale_date_str is not None:
            sale_date_str = parse_datetime(sale_date_str).isoformat().replace("T", " ")

        item["sale_datetime"] = sale_date_str
        item["sale_price"] = (
            response.xpath(
                '//tr[./td[text()="Sale Price"]]/td[@class="DataletData"]/text()'
            )
            .get("")
            .replace("$", "")
            .replace(",", "")
        )
        item["seller_1_name"] = (
            response.xpath(
                '//tr[./td[text()="Grantor"]]/td[@class="DataletData"]/text()'
            )
            .get("")
            .replace("\xa0", "")
        )
        item["buyer_1_name"] = (
            response.xpath(
                '//tr[./td[text()="Grantee"]]/td[@class="DataletData"]/text()'
            )
            .get("")
            .replace("\xa0", "")
        )
        item["seller_2_name"] = (
            response.xpath(
                '//tr[./td[text()="Grantor 2"]]/td[@class="DataletData"]/text()'
            )
            .get("")
            .replace("\xa0", "")
        )
        item["buyer_2_name"] = (
            response.xpath(
                '//tr[./td[text()="Grantee 2"]]/td[@class="DataletData"]/text()'
            )
            .get("")
            .replace("\xa0", "")
        )
        item["book"] = response.xpath(
            '//tr[./td[text()="Deed Book"]]/td[@class="DataletData"]/text()'
        ).get()
        item["page"] = response.xpath(
            '//tr[./td[text()="Deed Page"]]/td[@class="DataletData"]/text()'
        ).get()
        item["transfer_deed_type"] = response.xpath(
            '//tr[./td[text()="Instrument Type"]]/td[@class="DataletData"]/text()'
        ).get()
        item[
            "source_url"
        ] = "https://publicaccess.claytoncountyga.gov/search/advancedsearch.aspx?mode=advanced"

        appr_row = appr_rows.get(sale_date.year)
        if appr_row is not None:
            item["total_appraised_value"] = appr_row.get("Total", "").replace(",", "")
            item["land_appraised_value"] = appr_row.get("Land", "").replace(",", "")
            item["building_appraised_value"] = appr_row.get("Building", "").replace(
                ",", ""
            )
        else:
            item["total_appraised_value"] = None
            item["land_appraised_value"] = None
            item["building_appraised_value"] = None

        as_row = as_rows.get(sale_date.year)
        if as_row is not None:
            item["building_assessed_value"] = as_row.get("Buidling", "").replace(
                ",", ""
            )
            item["building_assessed_date"] = sale_date.year
            item["land_assessed_value"] = as_row.get("Land", "").replace(",", "")
            item["total_assessed_value"] = as_row.get("Total", "").replace(",", "")
        else:
            item["building_assessed_value"] = None
            item["building_assessed_value"] = None
            item["land_assessed_value"] = None
            item["total_assessed_value"] = None

        yield item

Again, nothing particularly fancy here. Just doing XPath queries and string manipulation to fill some more fields in the item object. We also access the assessed/appraised value dictionaries we created earlier to fill the relevant numbers for the same year as the sale (if possible).

But there’s more. We also have to traverse to next page, to get data for next sale.

        next_sale_link = response.xpath(
            '//a[./i[@class="icon-angle-right "]]/@href'
        ).get()
        if next_sale_link is not None:
            yield response.follow(
                next_sale_link,
                callback=self.parse_property_sales_page,
                dont_filter=True,
                meta={"item": item},
            )
            return

We extract the link to next sale and follow it. But why did we do return after yielding the new request? Well, that’s because at this point we merely want to go the page of next sale record. Further in the method we have code that does form submission to go the summary page of next parcel, so this entire shebang can be done again for next property:

        to_from_input_text = response.xpath(
            '//input[@name="DTLNavigator$txtFromTo"]/@value'
        ).get()
        idx, total = to_from_input_text.split(" of ")
        idx = int(idx)
        total = int(total)

        if idx == total:
            yield scrapy.Request(
                self.start_urls[-1],
                callback=self.parse_form_page,
                dont_filter=True,
                meta={"item": item},
            )
            return

        form_data = self.extract_form(response, '//form[@name="frmMain"]')
        del form_data["DTLNavigator$imageNext"]
        del form_data["DTLNavigator$imageLast"]
        form_data["DTLNavigator$imageNext.x"] = "0"  # XXX
        form_data["DTLNavigator$imageNext.y"] = str(idx + 1)
        form_data["hdMode"] = "DEK_PROFILE"
        logging.info(form_data)

        action = response.xpath('//form[@name="frmMain"]/@action').get()
        form_url = urljoin(response.url, action).replace(
            "mode=sales", "mode=dek_profile"
        )

        yield FormRequest(
            form_url,
            formdata=form_data,
            callback=self.parse_property_main_page,
            dont_filter=True,
        )

We take care to modify URL and hdMode field to make it go to the summary page, not the “Sales” page. We also address the possibility of the current parcel being the last in the list, which would warrant going to the next scheduled search space shard.

You can notice that requests tend to be instantiated with dont_filter flag. This is because site being scraped does not have unique URLs for many pages, which would trigger Scrapy deduplication mechanism needlessly, thus cancelling the scraping job before it was properly finished.

One more thing I would like to explain is how the scraping was parallelised to allow running multiple client sessions in parallel and retry the search shard scraping if needed. The spider has the following constructor method:

    def __init__(self, year=None, month=None, stats_filepath=None):
        super().__init__()

        self.stats_filepath = stats_filepath

        if year is not None and month is not None:
            year = int(year)
            month = int(month)
            self.shards = [(year, month)]
            return

        for year in range(self.start_year, datetime.today().year + 1):
            for month in range(1, 13):
                shard = (year, month)
                self.shards.append(shard)

If year and month values are provided through Scrapy CLI it will plan to scrape a single month worth of data. Otherwise, it will plan to scrape the entire data, one month at a time. How does this help us? Well, there is a helper script that will launch bunch of Scrapy processes in parallel and retry them if needed (based on some error conditions that may happen due to site instability):

#!/usr/bin/python3

import multiprocessing
import logging
import subprocess
import json
import os
import sys

spider_name = None


def run_scrapy_subprocess(year, month):
    csv_filename = "{}-{}-{}.csv".format(spider_name, year, month)
    if os.path.isfile(csv_filename):
        os.unlink(csv_filename)

    log_filename = "scrapy-{}-{}.log".format(year, month)
    if os.path.isfile(log_filename):
        os.unlink(log_filename)

    stats_filename = "stats-{}-{}.json".format(year, month)

    cmd = [
        "scrapy",
        "runspider",
        "-o",
        csv_filename,
        "--logfile",
        log_filename,
        "housingprices/spiders/{}.py".format(spider_name),
        "-a",
        "year={}".format(year),
        "-a",
        "month={}".format(month),
        "-a",
        "stats_filepath={}".format(stats_filename),
    ]

    print(cmd)

    result = subprocess.run(cmd)
    if result.returncode != 0:
        return False

    json_f = open(stats_filename, "r")
    stats = json.load(json_f)
    json_f.close()

    if stats.get("log_count/ERROR") is not None:
        return False

    if (
        stats.get("downloader/response_status_count/500") is not None
        and stats.get("downloader/response_status_count/500") > 8
    ):
        return False

    if (
        stats.get("downloader/response_status_count/501") is not None
        and stats.get("downloader/response_status_count/501") > 8
    ):
        return False

    return True


def perform_task(shard):
    year, month = shard
    print("Year: {}, month: {}".format(year, month))

    while True:
        success = run_scrapy_subprocess(year, month)
        if success:
            print("{} {} succeeded".format(year, month))
            break

        print("{} {} failed - retrying".format(year, month))


def main():
    global spider_name

    if len(sys.argv) != 2:
        print("Usage:")
        print("{} <spider_name>".format(sys.argv[0]))
        return

    spider_name = sys.argv[1]

    shards = []

    from_year = 1901
    to_year = 2022

    for year in range(from_year, to_year + 1):
        for month in range(1, 13):
            shard = (year, month)
            shards.append(shard)

    pool = multiprocessing.Pool(32)
    pool.map(perform_task, shards)


if __name__ == "__main__":
    main()

I was running this on VPS inside tmux session to perform the scraping. One thing you may notice is that it sets stats_filepath to file name unique to scraping job and read JSON from this file to check if scraping has succceeded. This requires some setup in Scrapy project. One thing we need is following piece of code to save the scraping stats when spider is done scraping:

# Based on:
# https://stackoverflow.com/questions/61402939/scrapy-how-to-save-crawling-statistics-to-json-file

from scrapy.statscollectors import StatsCollector
from scrapy.utils.serialize import ScrapyJSONEncoder


class MyStatsCollector(StatsCollector):
    def _persist_stats(self, stats, spider):
        encoder = ScrapyJSONEncoder()

        filename = "stats.json"
        if spider.stats_filepath is not None:
            filename = spider.stats_filepath

        out_f = open(filename, "w")
        data = encoder.encode(stats)
        out_f.write(data)
        out_f.close()

This has to be enabled in the settings as the linked Stack Overflow answer describes.

To sum it up, a complete Scrapy spider code is as follows:

import scrapy

from scrapy.http import FormRequest
from dateutil.parser import parse as parse_datetime

import calendar
from datetime import datetime, date
import logging
from urllib.parse import urljoin

from housingprices.items import SalesItem


class ClaytonSpider(scrapy.Spider):
    name = "clayton"
    allowed_domains = ["publicaccess.claytoncountyga.gov"]
    start_urls = [
        "https://publicaccess.claytoncountyga.gov/Search/Disclaimer.aspx?FromUrl=../search/advancedsearch.aspx?mode=advanced",
        "https://publicaccess.claytoncountyga.gov/search/advancedsearch.aspx?mode=advanced",
    ]
    start_year = 1939
    state = "GA"
    county = "CLAYTON"
    shards = []
    stats_filepath = None

    def __init__(self, year=None, month=None, stats_filepath=None):
        super().__init__()

        self.stats_filepath = stats_filepath

        if year is not None and month is not None:
            year = int(year)
            month = int(month)
            self.shards = [(year, month)]
            return

        for year in range(self.start_year, datetime.today().year + 1):
            for month in range(1, 13):
                shard = (year, month)
                self.shards.append(shard)

    def start_requests(self):
        yield scrapy.Request(self.start_urls[0], callback=self.parse_disclaimer_page)

    def parse_disclaimer_page(self, response):
        yield FormRequest.from_response(
            response,
            formxpath='//form[@name="Form1"]',
            clickdata={"name": "btAgree"},
            callback=self.parse_form_page,
        )

    def stringify_date(self, d):
        return "{:02d}/{:02d}/{:04d}".format(d.month, d.day, d.year)

    def extract_form(self, response, form_xpath):
        form_data = dict()

        for hidden_input in response.xpath(form_xpath).xpath(".//input"):
            name = hidden_input.attrib.get("name")
            if name is None:
                continue
            value = hidden_input.attrib.get("value")
            if value is None:
                value = ""

            form_data[name] = value

        return form_data

    def make_search_form_request(self, response, start_date, end_date):
        start_date_str = self.stringify_date(start_date)
        end_date_str = self.stringify_date(end_date)

        form_data = self.extract_form(response, '//form[@name="frmMain"]')

        form_data["hdCriteria"] = "salesdate|{}~{}".format(start_date_str, end_date_str)
        form_data["ctl01$cal1"] = start_date.isoformat()
        form_data["ctl01$cal1$dateInput"] = start_date_str
        form_data["txtCrit"] = start_date_str
        form_data["ctl01$cal2"] = end_date.isoformat()
        form_data["ctl01$cal2$dateInput"] = end_date_str
        form_data["txtCrit2"] = end_date_str
        form_data["PageNum"] = "1"
        form_data["PageSize"] = "1"
        form_data["hdCriteriaTypes"] = "N|N|C|C|C|N|C|C|N|D|N|N|C|C|C|N|N"
        form_data["sCriteria"] = "0"

        logging.debug(form_data)

        return FormRequest(
            response.url,
            formdata=form_data,
            callback=self.parse_search_result_list,
            dont_filter=True,
        )

    def parse_form_page(self, response):
        if len(self.shards) == 0:
            yield None
            return

        year, month = self.shards.pop()
        logging.info("Year: {}, month: {}".format(year, month))

        days_in_month = calendar.monthrange(year, month)[1]

        start_date = date(year=year, month=month, day=1)
        end_date = date(year=year, month=month, day=days_in_month)

        yield self.make_search_form_request(response, start_date, end_date)

    def parse_search_result_list(self, response):
        first_row = response.xpath('//tr[@class="SearchResults"][1]')
        if len(first_row) != 1:
            yield scrapy.Request(
                self.start_urls[-1], callback=self.parse_form_page, dont_filter=True
            )
            return

        logging.debug(first_row.xpath(".//text()").getall())

        onclick = first_row.attrib.get("onclick")
        rel_url = onclick.replace("javascript:selectSearchRow('", "").replace("')", "")

        form_data = self.extract_form(response, '//form[@name="frmMain"]')
        form_data["hdLink"] = rel_url
        form_data["hdAction"] = "Link"
        form_data["hdSelectAllChecked"] = "false"
        form_data["sCriteria"] = "0"
        logging.debug(form_data)

        action = response.xpath('//form[@name="frmMain"]/@action').get()
        form_url = urljoin(response.url, action)

        yield FormRequest(
            form_url,
            formdata=form_data,
            callback=self.parse_property_main_page,
            dont_filter=True,
        )

    def parse_property_main_page(self, response):
        parid = response.xpath('//input[@id="hdXPin"]/@value').get()
        property_street_address = (
            response.xpath('//td[@class="DataletHeaderBottom"][last()]/text()')
            .get("")
            .strip()
        )
        property_street_address = " ".join(
            property_street_address.split()
        )  # https://stackoverflow.com/a/1546251

        item = SalesItem()
        item["state"] = self.state
        item["property_id"] = parid
        item["property_street_address"] = property_street_address
        item["property_county"] = self.county
        item["property_type"] = response.xpath(
            '//tr[./td[contains(text(),"Property Class")]]/td[@class="DataletData"]/text()'
        ).get()

        residential_link = response.xpath(
            '//a[./span[text()="Residential"]]/@href'
        ).get()
        yield response.follow(
            residential_link,
            meta={"item": item},
            callback=self.parse_property_residential_page,
            dont_filter=True,
        )

    def parse_property_residential_page(self, response):
        item = response.meta.get("item")

        parid = response.xpath('//input[@id="hdXPin"]/@value').get()
        assert parid == item["property_id"]

        item["building_year_built"] = response.xpath(
            '//tr[./td[text()="Year Built"]]/td[@class="DataletData"]/text()'
        ).get()
        item["building_num_beds"] = response.xpath(
            '//tr[./td[text()="Bedrooms"]]/td[@class="DataletData"]/text()'
        ).get()
        item["building_num_baths"] = response.xpath(
            '//tr[./td[text()="Full Baths"]]/td[@class="DataletData"]/text()'
        ).get()

        value_history_link = response.xpath(
            '//a[./span[text()="Value History"]]/@href'
        ).get()
        yield response.follow(
            value_history_link,
            meta={"item": item},
            callback=self.parse_property_value_history_page,
            dont_filter=True,
        )

    def parse_property_value_history_page(self, response):
        item = response.meta.get("item")
        appr_rows = response.meta.get("appr_rows")
        as_rows = response.meta.get("as_rows")

        appr_rows = dict()
        appr_header = None

        appr_values_table = response.xpath('//table[@id="Appraised Values"]')
        if appr_values_table is not None:
            for row in appr_values_table.xpath("./tr"):
                row = row.xpath("./td/text()").getall()
                if appr_header is None:
                    appr_header = row
                    continue

                if len(row) == 0:
                    continue

                year = row[0]
                try:
                    year = int(year)
                except:
                    continue
                row = dict(zip(appr_header, row))
                appr_rows[year] = row

        logging.debug(appr_rows)

        as_rows = dict()
        as_header = None

        as_values_table = response.xpath('//table[@id="Assessed Values"]')
        if as_values_table is not None:
            for row in as_values_table.xpath("./tr"):
                row = row.xpath("./td/text()").getall()
                if as_header is None:
                    as_header = row
                    continue

                if len(row) == 0:
                    continue

                year = row[0]
                try:
                    year = int(year)
                except:
                    continue
                row = dict(zip(as_header, row))
                as_rows[year] = row

        logging.debug(as_rows)

        meta_dict = {"item": item, "appr_rows": appr_rows, "as_rows": as_rows}

        land_link = response.xpath('//a[./span[text()="Land"]]/@href').get()
        yield response.follow(
            land_link,
            meta=meta_dict,
            callback=self.parse_property_land_page,
            dont_filter=True,
        )

    def parse_property_land_page(self, response):
        item = response.meta.get("item")
        appr_rows = response.meta.get("appr_rows")
        as_rows = response.meta.get("as_rows")

        item["land_area_acres"] = (
            response.xpath('//tr[./td[text()="Acres"]]/td[@class="DataletData"]/text()')
            .get("")
            .replace("\xa0", "")
        )
        if item["land_area_acres"].startswith("."):
            item["land_area_acres"] = 0
        elif item["land_area_acres"] != "":
            item["land_area_acres"] = round(float(item["land_area_acres"]))
        item["land_area_sqft"] = (
            response.xpath(
                '//tr[./td[text()="Square Feet"]]/td[@class="DataletData"]/text()'
            )
            .get("")
            .replace("\xa0", "")
            .replace(",", "")
        )
        item["land_type"] = response.xpath(
            '//tr[./td[text()="Land Type"]]/td[@class="DataletData"]/text()'
        ).get()

        meta_dict = {"item": item, "appr_rows": appr_rows, "as_rows": as_rows}

        sales_link = response.xpath('//a[./span[text()="Sales"]]/@href').get()
        yield response.follow(
            sales_link,
            meta=meta_dict,
            callback=self.parse_property_sales_page,
            dont_filter=True,
        )

    def parse_property_sales_page(self, response):
        item = response.meta.get("item")
        appr_rows = response.meta.get("appr_rows")
        as_rows = response.meta.get("as_rows")

        sale_date_str = response.xpath(
            '//tr[./td[text()="Sale Date"]]/td[@class="DataletData"]/text()'
        ).get()
        if sale_date_str is not None:
            sale_date_str = parse_datetime(sale_date_str).isoformat().replace("T", " ")

        item["sale_datetime"] = sale_date_str
        item["sale_price"] = (
            response.xpath(
                '//tr[./td[text()="Sale Price"]]/td[@class="DataletData"]/text()'
            )
            .get("")
            .replace("$", "")
            .replace(",", "")
        )
        item["seller_1_name"] = (
            response.xpath(
                '//tr[./td[text()="Grantor"]]/td[@class="DataletData"]/text()'
            )
            .get("")
            .replace("\xa0", "")
        )
        item["buyer_1_name"] = (
            response.xpath(
                '//tr[./td[text()="Grantee"]]/td[@class="DataletData"]/text()'
            )
            .get("")
            .replace("\xa0", "")
        )
        item["seller_2_name"] = (
            response.xpath(
                '//tr[./td[text()="Grantor 2"]]/td[@class="DataletData"]/text()'
            )
            .get("")
            .replace("\xa0", "")
        )
        item["buyer_2_name"] = (
            response.xpath(
                '//tr[./td[text()="Grantee 2"]]/td[@class="DataletData"]/text()'
            )
            .get("")
            .replace("\xa0", "")
        )
        item["book"] = response.xpath(
            '//tr[./td[text()="Deed Book"]]/td[@class="DataletData"]/text()'
        ).get()
        item["page"] = response.xpath(
            '//tr[./td[text()="Deed Page"]]/td[@class="DataletData"]/text()'
        ).get()
        item["transfer_deed_type"] = response.xpath(
            '//tr[./td[text()="Instrument Type"]]/td[@class="DataletData"]/text()'
        ).get()
        item[
            "source_url"
        ] = "https://publicaccess.claytoncountyga.gov/search/advancedsearch.aspx?mode=advanced"

        appr_row = appr_rows.get(sale_date.year)
        if appr_row is not None:
            item["total_appraised_value"] = appr_row.get("Total", "").replace(",", "")
            item["land_appraised_value"] = appr_row.get("Land", "").replace(",", "")
            item["building_appraised_value"] = appr_row.get("Building", "").replace(
                ",", ""
            )
        else:
            item["total_appraised_value"] = None
            item["land_appraised_value"] = None
            item["building_appraised_value"] = None

        as_row = as_rows.get(sale_date.year)
        if as_row is not None:
            item["building_assessed_value"] = as_row.get("Buidling", "").replace(
                ",", ""
            )
            item["building_assessed_date"] = sale_date.year
            item["land_assessed_value"] = as_row.get("Land", "").replace(",", "")
            item["total_assessed_value"] = as_row.get("Total", "").replace(",", "")
        else:
            item["building_assessed_value"] = None
            item["building_assessed_value"] = None
            item["land_assessed_value"] = None
            item["total_assessed_value"] = None

        yield item

        next_sale_link = response.xpath(
            '//a[./i[@class="icon-angle-right "]]/@href'
        ).get()
        if next_sale_link is not None:
            yield response.follow(
                next_sale_link,
                callback=self.parse_property_sales_page,
                dont_filter=True,
                meta={"item": item},
            )
            return

        to_from_input_text = response.xpath(
            '//input[@name="DTLNavigator$txtFromTo"]/@value'
        ).get()
        idx, total = to_from_input_text.split(" of ")
        idx = int(idx)
        total = int(total)

        if idx == total:
            yield scrapy.Request(
                self.start_urls[-1],
                callback=self.parse_form_page,
                dont_filter=True,
                meta={"item": item},
            )
            return

        form_data = self.extract_form(response, '//form[@name="frmMain"]')
        del form_data["DTLNavigator$imageNext"]
        del form_data["DTLNavigator$imageLast"]
        form_data["DTLNavigator$imageNext.x"] = "0"  # XXX
        form_data["DTLNavigator$imageNext.y"] = str(idx + 1)
        form_data["hdMode"] = "DEK_PROFILE"
        logging.info(form_data)

        action = response.xpath('//form[@name="frmMain"]/@action').get()
        form_url = urljoin(response.url, action).replace(
            "mode=sales", "mode=dek_profile"
        )

        yield FormRequest(
            form_url,
            formdata=form_data,
            callback=self.parse_property_main_page,
            dont_filter=True,
        )

One last thing of interest are the following lines in HTML source code of the site being scraped:

<meta http-equiv='Description' content='iasWorld'>
<meta name='Copyright' content='Tyler Technologies Inc, Akanda Solutions, LLC'>

This points us to a product named iasWorld by a company called Tyler Technologies, Inc. That means there’s more sites like this and we can reuse the above code to scrape them (some modifications will be needed).

Making heuristics smarter with GPT-3

Sun, 09 Oct 2022 00:00:00 +0000

Sometimes data being scraped is a little fuzzy - easy to understand for human mind, but not strict and formal enough to be easily tractable by algorithms. This can get in a way of analysing the data or basing further automations off it. This is where heuristics come in. A heuristic is an inexact, but quick way to solve a problem. We will discuss how GPT-3 API can be used to implement heuristic techniques to deal with somewhat fuzzy data that might otherwise require a human to take a look into it.

Earlier this year I participated in data bounty that involved sourcing museum collection data and wrangling all the records into a single database schema. Some websites provided a great deal of museum collection data, but they also provided data on private art collections, libraries, universities, schools and other entitities that were neither art galleries nor museums, meaning that some data was out of scope. This presented a bit of a problem since some data needed to be scraped and submitted, but some had to be ignored based on institution name alone.

So what do we do? Well, given that GPT-3 was trained on massive amounts of text, it should be able to tell based on institution name if it is of interest for scraping. We just need to ask it properly. For example, we could give it a prompt:

Answer YES if "National Gallery of Art" is a gallery or museum. Answer NO otherwise.

To which it answers:

Yes

The part of the prompt that we keep changing is the substring between duoble quotes. We keep everything else the same. We also set the temperature gauge to 0 as we need no randomisation/“creativity” on AI part to answer this simple question.

Screenshot 1

Now that we got the prompt right we can use OpenAI client library to develop a Python script that will filter a scraped list of institutions to shortlist them for further scraping.

#!/usr/bin/python3

import csv
import os

import openai

openai.api_key = "[REDACTED]"

FIELDNAMES = ["name", "slug"]


def get_answer(institution_name):
    if "Archive" in institution_name:
        return "NO"

    if "Library" in institution_name or "Libraries" in institution_name:
        return "NO"

    if "Private Collection" in institution_name:
        return "NO"

    if "Museum" in institution_name:
        return "YES"

    if "Gallery" in institution_name:
        return "YES"

    response = openai.Completion.create(
        model="text-davinci-002",
        prompt='Answer YES if "{}" is a gallery or museum. Answer NO otherwise.'.format(
            institution_name
        ),
        temperature=0,
        max_tokens=4,
        top_p=1,
        frequency_penalty=0,
        presence_penalty=0,
    )

    answer = response.get("choices")[0].text.upper().strip()

    return answer


def main():
    in_f = open("orgs.csv", "r")
    csv_reader = csv.DictReader(in_f)

    out_f = open("filtered_orgs.csv", "w", encoding="utf-8")
    csv_writer = csv.DictWriter(out_f, fieldnames=FIELDNAMES, lineterminator="\n")
    csv_writer.writeheader()

    for row in csv_reader:
        name = row.get("name")
        answer = get_answer(name)
        print(name, answer)

        if answer == "YES":
            csv_writer.writerow(row)

    in_f.close()
    out_f.close()


if __name__ == "__main__":
    main()

To make things a little faster, we reject or approve some of the entries based on conclusive-enough substrings in the institution name. We only consult GPT-3 for the entries that remained.

Let us take a look into another example. Hurricane Electric provides a small web app to look up autonomous systems and IP ranges gives a company name. This might be of interest for bug bounty purposes. However, the result data is somewhat hairy. For example, searching for “Tesla Motors Inc.” also gives results related to “AD Aerodrom ‘Nikola Tesla’ Beograd”, “Tesla Comunicaciones SRL” and so on. Clearly the result set needs some filtering before it can be used for further work.

Screenshot 2

Like in our previous example, GPT-3 can be used to implement the heuristic. A preliminary prompt would something like:

Answer YES if "Tesla Motors, Inc." and "Tesla Comunicaciones SRL" are the same company. Answer NO otherwise.

Like in previous example, the code would perform string interpolation to generate a prompt for each entry in the result list.

Since we’re doing heuristics here we should not expect the answer to be 100% correct for each and every entry. Neither GPT-3 nor human intelligence can guarrantee that. However if we have a lot of fuzzy data to go through GPT-3 can do it far faster than people.

JavaScript AST manipulation with Babel: defeating string array mapping

Sat, 08 Oct 2022 00:00:00 +0000

One Javascript obfuscation technique that can be found in the wild is string array mapping. It entails gathering all the string constants from the code into an array and modifying the code so that string literals are accesssed by referencing the array at various indices instead of having string literals being used directly. Like with previously discussed obfuscation techniques we will be exploring it in isolation, but in real websites it will most likely be used in combination with another techniques.

Let us consider the following code:

console.log("Hello Venus");
console.log("Hello Earth");
console.log("Hello Mars");

Obfuscating this with Javascript Obfuscator “Move Strings” option turns it into the following obfuscated version:

var _0x3baf=["Hello Venus","log","Hello Earth","Hello Mars"];console[_0x3baf[1]](_0x3baf[0]);console[_0x3baf[1]](_0x3baf[2]);console[_0x3baf[1]](_0x3baf[3])

Screenshot 1

We have a single line consisting of multiple JS statements. Let us break this down for easier readability:

var _0x3baf=["Hello Venus","log","Hello Earth","Hello Mars"];
console[_0x3baf[1]](_0x3baf[0]);
console[_0x3baf[1]](_0x3baf[2]);
console[_0x3baf[1]](_0x3baf[3])

We can see that _0x3baf is an array of strings with all the string literals that were used in the code (including log as console.log is equivalent to console["log"]) and that is being referenced in further statements. Let us analyse this code in ASTExplorer.

Screenshot 2

Whenever the aforementioned string array is being used to get a string constant, we have a MemberExpression node in the tree with object instance variable being Identifier for the array name and property being NumericLiteral with index value. Of course, not every reference to an array will be related to string obfuscation, so we will have to check in our deobfuscation script if the array element at given index is indeed a string literal.

Like in previous posts, our code will have three parts:

Reading the JS code and parsing it into Babel AST data model.
Traversing the AST and modifying it using a visitor function.
Regenerating the code from AST.

In step 2 we will be using getBinding() method to get a binding of an array name. This will let us dig up an ArrayExpression node that has all the array elements so that we can perform an AST node substitution.

The code to deobfuscate this simple case of string array mapping is as follows:

const fs = require("fs");

const parser = require("@babel/parser");
const generate = require("@babel/generator").default;
const traverse = require("@babel/traverse").default;
const types = require("@babel/types");

let code = fs.readFileSync("obfuscated.js", "utf-8");

const ast = parser.parse(code);

traverse(ast, {
    MemberExpression: function(path) {
        if (!path.node.property) return;
        if (!types.isNumericLiteral(path.node.property)) return;

        let idx = path.node.property.value;

        let binding = path.scope.getBinding(path.node.object.name);
        if (!binding) return;
        
        if (types.isVariableDeclarator(binding.path.node)) {
            let array = binding.path.node.init;
            if (idx >= array.length) return;

            let member = array.elements[idx];

            if (types.isStringLiteral(member)) {
                path.replaceWith(member);
            }
        }
    }
});

fs.writeFileSync("clean.js", generate(ast).code);

We tie our visitor function to MemberExpression node type as that corresponds to parts of AST we want to modify. We make sure that property is indeed a numeric literal and take an array index from there. Next, we get the binding based on array name. On success, binding will point to a relevant VariableDeclarator node (through path and node) that points to the ArrayExpression node through init property. We have a reference to an array and we also have an index. We get the the array member. If it is indeed a string literal we perform the node substitution.

The deobfuscated code is as follows:

var _0x3baf = ["Hello Venus", "log", "Hello Earth", "Hello Mars"];
console["log"]("Hello Venus");
console["log"]("Hello Earth");
console["log"]("Hello Mars");

By working with Abstract Syntax Tree we were able to treat the code as data structure that can be manipulated to simplify it for undoing the obfuscation.

Active DNS recon techniques

Sat, 24 Sep 2022 00:00:00 +0000

We discussed some passive DNS recon techniques, but that’s only half of the story. There is also the active DNS reconnaissance that involves generating actual DNS requests to gather data. Active DNS recon does not rely on information footprints and secondary sources. Thus it enables us to get more up-to-date information on target systems.

So how do we send the DNS queries? We want to control what exact request we send and bypass DNS caching on the local system, so using getaddrinfo() is out of question. Simplest way is to use nslookup(1) - a standard POSIX tool for doing DNS queries:

$ nslookup hackerone.com 8.8.8.8  
Server:		8.8.8.8
Address:	8.8.8.8#53

Non-authoritative answer:
Name:	hackerone.com
Address: 104.16.99.52
Name:	hackerone.com
Address: 104.16.100.52

$ nslookup -query="MX" hackerone.com 8.8.8.8
Server:		8.8.8.8
Address:	8.8.8.8#53

Non-authoritative answer:
hackerone.com	mail exchanger = 30 aspmx2.googlemail.com.
hackerone.com	mail exchanger = 20 alt1.aspmx.l.google.com.
hackerone.com	mail exchanger = 10 aspmx.l.google.com.
hackerone.com	mail exchanger = 30 aspmx3.googlemail.com.
hackerone.com	mail exchanger = 20 alt2.aspmx.l.google.com.

Authoritative answers can be found from:

We can differentiate between successful and failed queries by checking the return code. It will be 0 on success on 1 on failure. That can be useful when checking a list of gueesses for subdomains in a shell script.

Another, more flexible tool to query DNS servers is dig(1). Depending on specifics of your exact system you may need to install it separately. It provides the output in a different, more machine readable form:

$ dig @8.8.8.8 hackerone.com

; <<>> DiG 9.10.6 <<>> @8.8.8.8 hackerone.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 16157
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;hackerone.com.			IN	A

;; ANSWER SECTION:
hackerone.com.		300	IN	A	104.16.100.52
hackerone.com.		300	IN	A	104.16.99.52

;; Query time: 55 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Fri Sep 23 17:34:16 +07 2022
;; MSG SIZE  rcvd: 74

$ dig @8.8.8.8 -t MX hackerone.com

; <<>> DiG 9.10.6 <<>> @8.8.8.8 -t MX hackerone.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 7903
;; flags: qr rd ra; QUERY: 1, ANSWER: 5, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;hackerone.com.			IN	MX

;; ANSWER SECTION:
hackerone.com.		300	IN	MX	30 aspmx2.googlemail.com.
hackerone.com.		300	IN	MX	20 alt1.aspmx.l.google.com.
hackerone.com.		300	IN	MX	10 aspmx.l.google.com.
hackerone.com.		300	IN	MX	30 aspmx3.googlemail.com.
hackerone.com.		300	IN	MX	20 alt2.aspmx.l.google.com.

;; Query time: 48 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Fri Sep 23 17:34:40 +07 2022
;; MSG SIZE  rcvd: 172

If you code in Python you can use dnspython module for performing DNS queries:

$ python3
Python 3.10.6 (main, Aug 30 2022, 04:58:14) [Clang 13.1.6 (clang-1316.0.21.2.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import dns.resolver
>>> resolver = dns.resolver.Resolver()
>>> resolver.nameservers = ['8.8.8.8']
>>> ans = resolver.resolve('hackerone.com')
>>> ans
<dns.resolver.Answer object at 0x1058cae30>
>>> ans.response
<DNS message, ID 55540>
>>> ans.response.answer
[<DNS hackerone.com. IN A RRset: [<104.16.100.52>, <104.16.99.52>]>]
>>> ans = resolver.resolve('hackerone.com', 'A')
>>> ans.response.answer
[<DNS hackerone.com. IN A RRset: [<104.16.100.52>, <104.16.99.52>]>]
>>> ans = resolver.resolve('hackerone.com', 'MX')
>>> ans.response.answer
[<DNS hackerone.com. IN MX RRset: [<30 aspmx2.googlemail.com.>, <20 alt1.aspmx.l.google.com.>, <10 aspmx.l.google.com.>, <30 aspmx3.googlemail.com.>, <20 alt2.aspmx.l.google.com.>]>]

We see that the data model of this library mirrors that of DNS protocol - there’s a object tree representing various layers/parts of a response.

We may want to perform a reverse DNS query that would give us a list of DNS names for a given IP address. For example, there might a single web server belonging to web hosting company that is hosting many websites, thus having multiple DNS addresses pointing to it. First we need to convert IPv4 address to domain-namish form by appending .in-addr.arpa. Next, we launch a PTR query:

$ dig @8.8.8.8 -t PTR 8.8.8.8.in-addr.arpa

; <<>> DiG 9.10.6 <<>> @8.8.8.8 -t PTR 8.8.8.8.in-addr.arpa
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 17913
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;8.8.8.8.in-addr.arpa.		IN	PTR

;; ANSWER SECTION:
8.8.8.8.in-addr.arpa.	18929	IN	PTR	dns.google.

;; Query time: 71 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Fri Sep 23 17:59:05 +07 2022
;; MSG SIZE  rcvd: 73

So given that we can launch DNS queries and they can yield some data on the target, how do we perform this at larger scale? One way is DNS bruteforcing that entails using a wordlist to generate possible subdomains (possibly by applying word permutations) and launching many DNS queries to check if they exist. Some tools to do this are:

Note that doing this will generate a lot of DNS requests that may crash your router, thus it is recommended to launch these tools from VPS. Furthermore, it may not be enough to use single DNS resolver like we did in earlier examples. When using multiple DNS resolvers, there’s a potential issue of data reliability. DNSValidator is a tool that takes a long list of DNS servers, runs test queries on them and over time narrows down the list to a subset of reliable servers.

Furthemore, if we know of one subdomain, e.g. dev.example.org we may suppose that numbered variants, such as dev1.example.org also exist. DNS alteration scanning is automated process of checking for these kind of subdomains. Altdns is open source tool that implements this technique.

So far we discussed launching DNS requests to learn about the target, but did not touch anything that interacts with the target. Yet another way to gather target subdomains is to crawl across the site and scrape them from the links or Javascript snippets. This would be fairly trivial to implement with Scrapy, but there are some ready-made tools to do this:

DNS zone transfer is mechanism to transfer DNS records between servers. It can be either authoritative that relies on AXFR sub-protocol or incremental that relies on IXFR. For recon purposes the feasibility of doing this relies on misconfigured access control on DNS server, which is not that common in modern world and should not be relied upon. There is a deliberately insecure DNS server that can be used to demonstrate this kind of vulnerability:

$ dig axfr @nsztm1.digi.ninja zonetransfer.me

; <<>> DiG 9.10.6 <<>> axfr @nsztm1.digi.ninja zonetransfer.me
; (1 server found)
;; global options: +cmd
zonetransfer.me.	7200	IN	SOA	nsztm1.digi.ninja. robin.digi.ninja. 2019100801 172800 900 1209600 3600
zonetransfer.me.	300	IN	HINFO	"Casio fx-700G" "Windows XP"
zonetransfer.me.	301	IN	TXT	"google-site-verification=tyP28J7JAUHA9fw2sHXMgcCC0I6XBmmoVi04VlMewxA"
zonetransfer.me.	7200	IN	MX	0 ASPMX.L.GOOGLE.COM.
zonetransfer.me.	7200	IN	MX	10 ALT1.ASPMX.L.GOOGLE.COM.
zonetransfer.me.	7200	IN	MX	10 ALT2.ASPMX.L.GOOGLE.COM.
zonetransfer.me.	7200	IN	MX	20 ASPMX2.GOOGLEMAIL.COM.
zonetransfer.me.	7200	IN	MX	20 ASPMX3.GOOGLEMAIL.COM.
zonetransfer.me.	7200	IN	MX	20 ASPMX4.GOOGLEMAIL.COM.
zonetransfer.me.	7200	IN	MX	20 ASPMX5.GOOGLEMAIL.COM.
zonetransfer.me.	7200	IN	A	5.196.105.14
zonetransfer.me.	7200	IN	NS	nsztm1.digi.ninja.
zonetransfer.me.	7200	IN	NS	nsztm2.digi.ninja.
_acme-challenge.zonetransfer.me. 301 IN	TXT	"6Oa05hbUJ9xSsvYy7pApQvwCUSSGgxvrbdizjePEsZI"
_sip._tcp.zonetransfer.me. 14000 IN	SRV	0 0 5060 www.zonetransfer.me.
14.105.196.5.IN-ADDR.ARPA.zonetransfer.me. 7200	IN PTR www.zonetransfer.me.
asfdbauthdns.zonetransfer.me. 7900 IN	AFSDB	1 asfdbbox.zonetransfer.me.
asfdbbox.zonetransfer.me. 7200	IN	A	127.0.0.1
asfdbvolume.zonetransfer.me. 7800 IN	AFSDB	1 asfdbbox.zonetransfer.me.
canberra-office.zonetransfer.me. 7200 IN A	202.14.81.230
cmdexec.zonetransfer.me. 300	IN	TXT	"; ls"
contact.zonetransfer.me. 2592000 IN	TXT	"Remember to call or email Pippa on +44 123 4567890 or pippa@zonetransfer.me when making DNS changes"
dc-office.zonetransfer.me. 7200	IN	A	143.228.181.132
deadbeef.zonetransfer.me. 7201	IN	AAAA	dead:beaf::
dr.zonetransfer.me.	300	IN	LOC	53 20 56.558 N 1 38 33.526 W 0.00m 1m 10000m 10m
DZC.zonetransfer.me.	7200	IN	TXT	"AbCdEfG"
email.zonetransfer.me.	2222	IN	NAPTR	1 1 "P" "E2U+email" "" email.zonetransfer.me.zonetransfer.me.
email.zonetransfer.me.	7200	IN	A	74.125.206.26
Hello.zonetransfer.me.	7200	IN	TXT	"Hi to Josh and all his class"
home.zonetransfer.me.	7200	IN	A	127.0.0.1
Info.zonetransfer.me.	7200	IN	TXT	"ZoneTransfer.me service provided by Robin Wood - robin@digi.ninja. See http://digi.ninja/projects/zonetransferme.php for more information."
internal.zonetransfer.me. 300	IN	NS	intns1.zonetransfer.me.
internal.zonetransfer.me. 300	IN	NS	intns2.zonetransfer.me.
intns1.zonetransfer.me.	300	IN	A	81.4.108.41
intns2.zonetransfer.me.	300	IN	A	167.88.42.94
office.zonetransfer.me.	7200	IN	A	4.23.39.254
ipv6actnow.org.zonetransfer.me.	7200 IN	AAAA	2001:67c:2e8:11::c100:1332
owa.zonetransfer.me.	7200	IN	A	207.46.197.32
robinwood.zonetransfer.me. 302	IN	TXT	"Robin Wood"
rp.zonetransfer.me.	321	IN	RP	robin.zonetransfer.me. robinwood.zonetransfer.me.
sip.zonetransfer.me.	3333	IN	NAPTR	2 3 "P" "E2U+sip" "!^.*$!sip:customer-service@zonetransfer.me!" .
sqli.zonetransfer.me.	300	IN	TXT	"' or 1=1 --"
sshock.zonetransfer.me.	7200	IN	TXT	"() { :]}; echo ShellShocked"
staging.zonetransfer.me. 7200	IN	CNAME	www.sydneyoperahouse.com.
alltcpportsopen.firewall.test.zonetransfer.me. 301 IN A	127.0.0.1
testing.zonetransfer.me. 301	IN	CNAME	www.zonetransfer.me.
vpn.zonetransfer.me.	4000	IN	A	174.36.59.154
www.zonetransfer.me.	7200	IN	A	5.196.105.14
xss.zonetransfer.me.	300	IN	TXT	"'><script>alert('Boo')</script>"
zonetransfer.me.	7200	IN	SOA	nsztm1.digi.ninja. robin.digi.ninja. 2019100801 172800 900 1209600 3600
;; Query time: 232 msec
;; SERVER: 81.4.108.41#53(81.4.108.41)
;; WHEN: Fri Sep 23 18:53:56 +07 2022
;; XFR size: 50 records (messages 1, bytes 1994)

We see that all records have been dumped without us needing to query them one-by-one.

If DNSSEC is deployed, NSEC records would exist and conveniently provide subdomains for us. Each NSEC points to next subdomain in DNS zone (a portion of DNS namespace managed by single admin/organisation) in a way that forms a linked list. Traversal of this list gives us a list of subdomains. This is known as DNS zone walking.

Some tools that can automate this technique:

DNSrecon
dnssecwalk from thc-ipv6 toolkit.

Passive DNS recon techniques

Sat, 17 Sep 2022 00:00:00 +0000

DNS is a network protocol and a distributed system that maps human-readable domain names to IP addresses. It can be thought of as a big phone book for internet hosts. Passive DNS recon is activity of using various data sources to map out a footprint of site or organisation without ever directly accessing the target or launching DNS requests. That’s basically doing OSINT on DNS names and records that are linked to the target. Passive DNS recon is not limited to the current state of DNS records, but also covers historical data and temporal analysis.

Passive DNS recon can be done for various purposes, such as:

Investigating security incidents.
Detecting shadow IT, misconfigurations and legacy/forgotten servers that might be insecure.
Mapping out attack surface for the organisation.
Detecting dangling subdomains linked to third party services that may provide opportunities for subdomain takeovers, perhaps leading to a bug bounty payment.
Detecting third party SaaS and cloud services being used by the target.
Gaining insights on business activities of competitors - for example, a SaaS company might have customer-specific subdomains.
Finding origin server to scrape the site in a way that bypasses countermeasures implemented by content delivery network.

Let us go through some of the ways one can perform passive DNS recon, assuming we have one or more seed domains to start from.

One way to do it is search engine dorking. We use site: operator with our seed domain to find some results from the site. Some of the results will probably be from subdomains. You exclude these subdomains from search results by using -site: thingy to get some more subdomains:

site:tiktok.com
site:tiktok.com -site:effecthouse.tiktok.com -site:business.tiktok.com -site:www.tiktok.com -site:newsroom.tiktok.com
site:tiktok.com -site:effecthouse.tiktok.com -site:business.tiktok.com -site:www.tiktok.com -site:newsroom.tiktok.com -site:vm.tiktok.com -site:seller-vn.tiktok.com -site:careers.tiktok.com -site:ads.tiktok.com
...

Screenshot 1 Screenshot 2

This is performed until there are no more new subdomains. There’s recon-ng module based on this trick to automatically scrape subdomains from Google SERPs. One can also use Bright Data Search Engine Crawler or similar product to outsource search engine scraping via API. Furthermore, Bing and other search engines can also be used for finding subdomains.

Github is another source for finding subdomains. Merely searching for seed domain in Github search (which can also be done via API) is likely to get you some subdomains that can be extracted with little string processing.

Screenshot 3

Yet another source is Internet Archive/Wayback machine. It does not give you subdomains directly, but allows searching for URLs with wildcard queries:

$ curl "http://web.archive.org/cdx/search/cdx?url=*.hackerone.com/*&output=json&collapse=urlkey" > urls.json

Subdomains can be trivially extracted from these URLs.

X.509 certificates can be mined to extract subdomains. To search through a lot of certificates, one can use certificate transparency tool such as crt.sh (it also exposes PostgreSQL database that you can connect to with psql(1): psql -h crt.sh -p 5432 -U guest certwatch) or query Facebook’s Certificate Transparency API.

Screenshot 4

Some other sources for finding subdomains include:

Since DNS data is relevant to security, many companies in the world gather it for you and provide it for your analysis via some web UI or API. Some of these companies are:

Censys
Shodan
SecurityTrails
IntelligenceX
RiskIQ
ZoomEye

To gather as much subdomains as possible, one needs to have automated way to query many data sources. Two most prominent tools to do that are:

Amass from OWASP.
Subfinder from Project Discovery.

Chaos is a project that provides subdomain datasets for companies running public bounty program (gathered through active enumeration that is outside the scope of this post).

DNSdumpster is a free domain research tool one can use to find subdomains and DNS records. It also provides a nice graphical visualisation on links between records.

Screenshot 5 Screenshot 6

HTTPX: modern Python module for doing HTTP(S) requests

Fri, 16 Sep 2022 00:00:00 +0000

Python requests module is a well established open source library for doing HTTP requests that is widely used in web scraping and other fields. However it has some limitations. At the time of writing, requests module only supports HTTP/1.1 yet significant fraction of sites are supporting more modern, faster HTTP/2 protocol and there is no support for asynchronous communication in requests module.

HTTPX is a newer, more modern Python module that addresses some of the limitations of requests module (not to be confused with another httpx that is CLI tool for probing HTTP servers). It can be installed as httpx through PIP. Furthermore, one can install the following optional dependencies:

h2 for HTTP/2 support (through httpx[http2])
socksio for SOCKS proxy support (through httpx[socks])
click for optional CLI tool (through httpx[cli])
rich for making the optional CLI tool prettier
brotli or brotlicffi for supporting Cloudflare’s Brotli compression (through httpx[brotli])

HTTPX is largely, but not entirely a drop-in replacement for the aforementioned requests module. There are some differences, such as:

It does not follow redirects automatically by default (we need to set pass follow_redirects=True).
.url on response object is URL object, not a string
Keys of proxy dictionary are http:// and https://, not http and https.
All request have timeout set them by default to avoid things getting stuck.

Let us launch Python REPL and make a simple HTTPS request.

$ python3
Python 3.10.6 (main, Aug 30 2022, 04:58:14) [Clang 13.1.6 (clang-1316.0.21.2.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import httpx
>>> resp = httpx.get("https://api.github.com/users/facebook-github-bot/events/public")
>>> resp
<Response [200 OK]>
>>> resp.status_code
200
>>> resp.text[:100]
'[{"id":"24020632984","type":"IssueCommentEvent","actor":{"id":6422482,"login":"facebook-github-bot",'
>>> type(resp.json())
<class 'list'>
>>> resp.json()[0]
{'id': '24020632984', 'type': 'IssueCommentEvent', 'actor': {'id': 6422482, 'login': 'facebook-github-bot', 'display_login': 'facebook-github-bot', 'gravatar_id': '', 'url': 'https://api.github.com/users/facebook-github-bot', 'avatar_url': 'https://avatars.githubusercontent.com/u/6422482?'}, 'repo': {'id': 392055467, 'name': 'facebookresearch/fbpcs', 'url': 'https://api.github.com/repos/facebookresearch/fbpcs'}, 'payload': {'action': 'created', 'issue': {'url': 'https://api.github.com/repos/facebookresearch/fbpcs/issues/432', 'repository_url': 'https://api.github.com/repos/facebookresearch/fbpcs', 'labels_url': 'https://api.github.com/repos/facebookresearch/fbpcs/issues/432/labels{/name}', 'comments_url': 'https://api.github.com/repos/facebookresearch/fbpcs/issues/432/comments', 'events_url': 'https://api.github.com/repos/facebookresearch/fbpcs/issues/432/events', 'html_url': 'https://github.com/facebookresearch/fbpcs/pull/432', 'id': 1070015172, 'node_id': 'PR_kwDOF15Kq84vVQRm', 'number': 432, 'title': 'update graphapi version', 'user': {'login': 'benliugithub', 'id': 90293689, 'node_id': 'MDQ6VXNlcjkwMjkzNjg5', 'avatar_url': 'https://avatars.githubusercontent.com/u/90293689?v=4', 'gravatar_id': '', 'url': 'https://api.github.com/users/benliugithub', 'html_url': 'https://github.com/benliugithub', 'followers_url': 'https://api.github.com/users/benliugithub/followers', 'following_url': 'https://api.github.com/users/benliugithub/following{/other_user}', 'gists_url': 'https://api.github.com/users/benliugithub/gists{/gist_id}', 'starred_url': 'https://api.github.com/users/benliugithub/starred{/owner}{/repo}', 'subscriptions_url': 'https://api.github.com/users/benliugithub/subscriptions', 'organizations_url': 'https://api.github.com/users/benliugithub/orgs', 'repos_url': 'https://api.github.com/users/benliugithub/repos', 'events_url': 'https://api.github.com/users/benliugithub/events{/privacy}', 'received_events_url': 'https://api.github.com/users/benliugithub/received_events', 'type': 'User', 'site_admin': False}, 'labels': [{'id': 3223376167, 'node_id': 'MDU6TGFiZWwzMjIzMzc2MTY3', 'url': 'https://api.github.com/repos/facebookresearch/fbpcs/labels/CLA%20Signed', 'name': 'CLA Signed', 'color': '009900', 'default': False, 'description': 'This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.'}, {'id': 3251191745, 'node_id': 'MDU6TGFiZWwzMjUxMTkxNzQ1', 'url': 'https://api.github.com/repos/facebookresearch/fbpcs/labels/fb-exported', 'name': 'fb-exported', 'color': 'ededed', 'default': False, 'description': None}], 'state': 'open', 'locked': False, 'assignee': None, 'assignees': [], 'milestone': None, 'comments': 1, 'created_at': '2021-12-02T21:54:32Z', 'updated_at': '2022-09-15T07:20:13Z', 'closed_at': None, 'author_association': 'NONE', 'active_lock_reason': None, 'draft': False, 'pull_request': {'url': 'https://api.github.com/repos/facebookresearch/fbpcs/pulls/432', 'html_url': 'https://github.com/facebookresearch/fbpcs/pull/432', 'diff_url': 'https://github.com/facebookresearch/fbpcs/pull/432.diff', 'patch_url': 'https://github.com/facebookresearch/fbpcs/pull/432.patch', 'merged_at': None}, 'body': 'Summary: As title\n\nReviewed By: leegross\n\nDifferential Revision: D32811061\n\n', 'reactions': {'url': 'https://api.github.com/repos/facebookresearch/fbpcs/issues/432/reactions', 'total_count': 0, '+1': 0, '-1': 0, 'laugh': 0, 'hooray': 0, 'confused': 0, 'heart': 0, 'rocket': 0, 'eyes': 0}, 'timeline_url': 'https://api.github.com/repos/facebookresearch/fbpcs/issues/432/timeline', 'performed_via_github_app': None, 'state_reason': None}, 'comment': {'url': 'https://api.github.com/repos/facebookresearch/fbpcs/issues/comments/1247684175', 'html_url': 'https://github.com/facebookresearch/fbpcs/pull/432#issuecomment-1247684175', 'issue_url': 'https://api.github.com/repos/facebookresearch/fbpcs/issues/432', 'id': 1247684175, 'node_id': 'IC_kwDOF15Kq85KXiZP', 'user': {'login': 'facebook-github-bot', 'id': 6422482, 'node_id': 'MDQ6VXNlcjY0MjI0ODI=', 'avatar_url': 'https://avatars.githubusercontent.com/u/6422482?v=4', 'gravatar_id': '', 'url': 'https://api.github.com/users/facebook-github-bot', 'html_url': 'https://github.com/facebook-github-bot', 'followers_url': 'https://api.github.com/users/facebook-github-bot/followers', 'following_url': 'https://api.github.com/users/facebook-github-bot/following{/other_user}', 'gists_url': 'https://api.github.com/users/facebook-github-bot/gists{/gist_id}', 'starred_url': 'https://api.github.com/users/facebook-github-bot/starred{/owner}{/repo}', 'subscriptions_url': 'https://api.github.com/users/facebook-github-bot/subscriptions', 'organizations_url': 'https://api.github.com/users/facebook-github-bot/orgs', 'repos_url': 'https://api.github.com/users/facebook-github-bot/repos', 'events_url': 'https://api.github.com/users/facebook-github-bot/events{/privacy}', 'received_events_url': 'https://api.github.com/users/facebook-github-bot/received_events', 'type': 'User', 'site_admin': False}, 'created_at': '2022-09-15T07:20:13Z', 'updated_at': '2022-09-15T07:20:13Z', 'author_association': 'CONTRIBUTOR', 'body': 'Hi @benliugithub! \n\nThank you for your pull request. \n\nWe **require** contributors to sign our **Contributor License Agreement**, and yours needs attention.\n\nYou currently have a record in our system, but the CLA is no longer valid, and will need to be **resubmitted**.\n\n# Process\n\nIn order for us to review and merge your suggested changes, please sign at <https://code.facebook.com/cla>. **If you are contributing on behalf of someone else (eg your employer)**, the individual CLA may not be sufficient and your employer may need to sign the corporate CLA.\n\nOnce the CLA is signed, our tooling will perform checks and validations. Afterwards, the **pull request will be tagged** with `CLA signed`. The tagging process may take up to 1 hour after signing. Please give it that time before contacting us about it.\n\nIf you have received this in error or have any questions, please contact us at [cla@fb.com](mailto:cla@fb.com?subject=CLA%20for%20facebookresearch%2Ffbpcs%20%23432). Thanks!', 'reactions': {'url': 'https://api.github.com/repos/facebookresearch/fbpcs/issues/comments/1247684175/reactions', 'total_count': 0, '+1': 0, '-1': 0, 'laugh': 0, 'hooray': 0, 'confused': 0, 'heart': 0, 'rocket': 0, 'eyes': 0}, 'performed_via_github_app': None}}, 'public': True, 'created_at': '2022-09-15T07:20:14Z', 'org': {'id': 16943930, 'login': 'facebookresearch', 'gravatar_id': '', 'url': 'https://api.github.com/orgs/facebookresearch', 'avatar_url': 'https://avatars.githubusercontent.com/u/16943930?'}}

This was HTTP/1.1 request. But what if we wanted to launch a HTTP/2 request? Assuming httpx[http2] is installed, we would need instantiate a client object with http2 keyword argument set to true:

>>> client = httpx.Client(http2=True)
>>> client
<httpx.Client object at 0x105f3a2c0>
>>> resp = client.get("https://api.github.com/users/facebook-github-bot/events/public")
>>> resp
<Response [200 OK]>
>>> resp.text[:100]
'[{"id":"24020972057","type":"PullRequestEvent","actor":{"id":6422482,"login":"facebook-github-bot","'
>>> resp.http_version
'HTTP/2'

The client object in HTTPX is largely equivalent to requests.Session: it can manage cookies between requests, reuse headers, proxy URLs and other settings. Furthermore, HTTPX client implements HTTP connection pooling for improved performance. This entails refraining from closing TCP connections to remote servers after getting response and keeping them available to save time when new request is available. The client object can be used as context manager, but if we wanted to explicitly close all connection in the pool we could do so by calling close() method.

If we want to benefit from async I/O we can use httpx.AsyncClient instead of httpx.Client:

$ python3 -m asyncio
asyncio REPL 3.10.6 (main, Aug 30 2022, 04:58:14) [Clang 13.1.6 (clang-1316.0.21.2.5)] on darwin
Use "await" directly instead of "asyncio.run()".
Type "help", "copyright", "credits" or "license" for more information.
>>> import asyncio
>>> import httpx
>>> async with httpx.AsyncClient() as client:
...     resp = await client.get("https://instagram.com")
... 
>>> resp
<Response [301 Moved Permanently]>

If you have installed the optional httpx[cli] submodule, you can use the CLI tool that is developed with the library:

$ httpx --download fb_bot.json https://api.github.com/users/facebook-github-bot/events/public
HTTP/1.1 200 OK
Server: GitHub.com
Date: Thu, 15 Sep 2022 07:53:21 GMT
Content-Type: application/json; charset=utf-8
Cache-Control: public, max-age=60, s-maxage=60
Vary: Accept, Accept-Encoding, Accept, X-Requested-With
ETag: W/"0d377aecaac2a62f7662fc84bfb6ca33a1e6372d14da1fa4f89a3a39f02e8191"
Last-Modified: Thu, 15 Sep 2022 07:44:01 GMT
X-Poll-Interval: 60
X-GitHub-Media-Type: github.v3; format=json
Link: <https://api.github.com/user/6422482/events/public?page=2>; rel="next", <https://api.github.com/user/6422482/events/public?page=8>; rel="last"
Access-Control-Expose-Headers: ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Used, X-RateLimit-Resource, X-RateLimit-Reset, X-OAuth-Scopes, 
X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, X-GitHub-SSO, X-GitHub-Request-Id, Deprecation, Sunset
Access-Control-Allow-Origin: *
Strict-Transport-Security: max-age=31536000; includeSubdomains; preload
X-Frame-Options: deny
X-Content-Type-Options: nosniff
X-XSS-Protection: 0
Referrer-Policy: origin-when-cross-origin, strict-origin-when-cross-origin
Content-Security-Policy: default-src 'none'
Content-Encoding: gzip
X-RateLimit-Limit: 60
X-RateLimit-Remaining: 52
X-RateLimit-Reset: 1663229406
X-RateLimit-Resource: core
X-RateLimit-Used: 8
Accept-Ranges: bytes
Transfer-Encoding: chunked
X-GitHub-Request-Id: EA77:1A4B:39239D:3B3337:6322D9F0


Downloading fb_bot.json   0% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 16,286/0 bytes ?
$ file fb_bot.json 
fb_bot.json: JSON data

HTTPX heeds the following environment variables:

HTTPX_LOG_LEVEL - enables extra logging for debugging purposes (can be debug or trace)
SSLKEYLOGFILE - path for saving TLS handshake secrets, so that TLS connections can be decrypted in Wireshark.
HTTP_PROXY and HTTPS_PROXY - proxy URLs to be used for HTTP and HTTPS requests.
ALL_PROXY - proxy URL to be used for all requests.
NO_PROXY - comma-separated list of hostnames and URLs that should be accessed without proxying.
NETRC - a path to a .netrc file.
SSL_CERT_FILE - path to custom CA certificate (.crt file). This may be useful when working with proxies that intercept TLS connections.
SSL_CERT_DIR - X.509 certificate directory path. This directory is expected to follow the OpenSSL layout.

Recon-ng: modular framework for OSINT automation

Wed, 07 Sep 2022 00:00:00 +0000

OSINT is collection and analysis of information from public sources. Nowadays it can largely be automated through web scraping and API integrations. Recon-ng is open source OSINT framework that features a modular architecture. Each module can be thought as a pluggable piece of code that can be loaded on as-needed basis. Most modules are API integrations and scrapers of data sources. Some deal with report generation and other auxillary tasks. Since recon-ng was developed by and for infosec people, the user interface resembles that of Metasploit Framework, thus making the learning curve easier to people who are working on the offensive side of cybersecurity.

Some security-centric Linux distributions ship recon-ng on their package managers. Furthermore, macOS users can install it through Homebrew. If none of these options are applicable one can simply run it from source code in a fairly vanilla Python dev environment.

The recon-ng script gives us a command line for running OSINT tasks:

$ recon-ng

    _/_/_/    _/_/_/_/    _/_/_/    _/_/_/    _/      _/            _/      _/    _/_/_/
   _/    _/  _/        _/        _/      _/  _/_/    _/            _/_/    _/  _/       
  _/_/_/    _/_/_/    _/        _/      _/  _/  _/  _/  _/_/_/_/  _/  _/  _/  _/  _/_/_/
 _/    _/  _/        _/        _/      _/  _/    _/_/            _/    _/_/  _/      _/ 
_/    _/  _/_/_/_/    _/_/_/    _/_/_/    _/      _/            _/      _/    _/_/_/    


                                          /\
                                         / \\ /\
    Sponsored by...               /\  /\/  \\V  \/\
                                 / \\/ // \\\\\ \\ \/\
                                // // BLACK HILLS \/ \\
                               www.blackhillsinfosec.com

                  ____   ____   ____   ____ _____ _  ____   ____  ____
                 |____] | ___/ |____| |       |   | |____  |____ |
                 |      |   \_ |    | |____   |   |  ____| |____ |____
                                   www.practisec.com

                      [recon-ng v5.1.2, Tim Tomes (@lanmaster53)]                       

[85] Recon modules
[13] Disabled modules
[8]  Reporting modules
[4]  Import modules
[2]  Exploitation modules
[2]  Discovery modules

[recon-ng][default] >

At this point it is likely to print bunch of errors about missing API keys and some Python modules you are advised to install. It is okay to disregard these when starting. We still have stuff to try out at this point.

Let us get familiar with the concept of workspaces. In recon-ng, a workspace is basically a project that holds together various pieces of data you collect. When you launch recon-ng you will be assigned a default workspace. Let us switch create new workspace and switch to it:

[recon-ng][default] > workspaces create test
[recon-ng][test] >

workspaces list command lists all the available workspaces:

[recon-ng][test] > workspaces list

  +----------------------------------+
  | Workspaces |       Modified      |
  +----------------------------------+
  | default    | 2022-09-05 11:30:15 |
  | test       | 2022-09-05 11:55:27 |
  +----------------------------------+

If we wanted to go back to default workspaces, we can say workspaces load default. To remove a workspace we no longer need we can say workspaces remove.

Like mentioned before, a recon-ng module is pluggable piece of code that can loaded when needed. All the modules are stored in a separate git repo called recon-ng-marketplace. Run marketplace install all to download them into a local installation. When that is done, running marketplace search will give you a table with available modules. Running modules list will give you list of modules by category. To search modules by keyword, we can run these commands with the keyword:

[recon-ng][test] > modules search domain
[*] Searching installed modules for 'domain'...

  Recon
  -----
    recon/companies-domains/censys_subdomains
    recon/companies-domains/pen
    recon/companies-domains/viewdns_reverse_whois
    recon/companies-domains/whoxy_dns
    recon/contacts-domains/migrate_contacts
    recon/domains-companies/pen
    recon/domains-companies/whoxy_whois
    recon/domains-contacts/hunter_io
    recon/domains-contacts/pen
    recon/domains-contacts/pgp_search
    recon/domains-contacts/whois_pocs
    recon/domains-contacts/wikileaker
    recon/domains-credentials/pwnedlist/api_usage
    recon/domains-credentials/pwnedlist/domain_ispwned
    recon/domains-credentials/pwnedlist/leak_lookup
    recon/domains-credentials/pwnedlist/leaks_dump
    recon/domains-domains/brute_suffix
    recon/domains-hosts/binaryedge
    recon/domains-hosts/bing_domain_api
    recon/domains-hosts/bing_domain_web
    recon/domains-hosts/brute_hosts
    recon/domains-hosts/builtwith
    recon/domains-hosts/certificate_transparency
    recon/domains-hosts/google_site_web
    recon/domains-hosts/hackertarget
    recon/domains-hosts/mx_spf_ip
    recon/domains-hosts/netcraft
    recon/domains-hosts/shodan_hostname
    recon/domains-hosts/spyse_subdomains
    recon/domains-hosts/ssl_san
    recon/domains-hosts/threatcrowd
    recon/domains-hosts/threatminer
    recon/domains-vulnerabilities/ghdb
    recon/domains-vulnerabilities/xssed
    recon/hosts-domains/migrate_hosts

[recon-ng][test] > marketplace search domain
[*] Searching module index for 'domain'...

  +-----------------------------------------------------------------------------------------------+
  |                        Path                        | Version |   Status  |  Updated   | D | K |
  +-----------------------------------------------------------------------------------------------+
  | discovery/info_disclosure/cache_snoop              | 1.1     | installed | 2020-10-13 |   |   |
  | recon/companies-domains/censys_subdomains          | 2.0     | installed | 2021-05-10 | * | * |
  | recon/companies-domains/pen                        | 1.1     | installed | 2019-10-15 |   |   |
  | recon/companies-domains/viewdns_reverse_whois      | 1.1     | installed | 2021-08-24 |   |   |
  | recon/companies-domains/whoxy_dns                  | 1.1     | installed | 2020-06-17 |   | * |
  | recon/companies-hosts/censys_tls_subjects          | 2.0     | disabled  | 2021-05-11 | * | * |
  | recon/contacts-domains/migrate_contacts            | 1.1     | installed | 2020-05-17 |   |   |
  | recon/domains-companies/censys_companies           | 2.0     | disabled  | 2021-05-10 | * | * |
  | recon/domains-companies/pen                        | 1.1     | installed | 2019-10-15 |   |   |
  | recon/domains-companies/whoxy_whois                | 1.1     | installed | 2020-06-24 |   | * |
  | recon/domains-contacts/hunter_io                   | 1.3     | installed | 2020-04-14 |   | * |
  | recon/domains-contacts/metacrawler                 | 1.1     | disabled  | 2019-06-24 | * |   |
  | recon/domains-contacts/pen                         | 1.1     | installed | 2019-10-15 |   |   |
  | recon/domains-contacts/pgp_search                  | 1.4     | installed | 2019-10-16 |   |   |
  | recon/domains-contacts/whois_pocs                  | 1.0     | installed | 2019-06-24 |   |   |
  | recon/domains-contacts/wikileaker                  | 1.0     | installed | 2020-04-08 |   |   |
  | recon/domains-credentials/pwnedlist/account_creds  | 1.0     | disabled  | 2019-06-24 | * | * |
  | recon/domains-credentials/pwnedlist/api_usage      | 1.0     | installed | 2019-06-24 |   | * |
  | recon/domains-credentials/pwnedlist/domain_creds   | 1.0     | disabled  | 2019-06-24 | * | * |
  | recon/domains-credentials/pwnedlist/domain_ispwned | 1.0     | installed | 2019-06-24 |   | * |
  | recon/domains-credentials/pwnedlist/leak_lookup    | 1.0     | installed | 2019-06-24 |   |   |
  | recon/domains-credentials/pwnedlist/leaks_dump     | 1.0     | installed | 2019-06-24 |   | * |
  | recon/domains-domains/brute_suffix                 | 1.1     | installed | 2020-05-17 |   |   |
  | recon/domains-hosts/binaryedge                     | 1.2     | installed | 2020-06-18 |   | * |
  | recon/domains-hosts/bing_domain_api                | 1.0     | installed | 2019-06-24 |   | * |
  | recon/domains-hosts/bing_domain_web                | 1.1     | installed | 2019-07-04 |   |   |
  | recon/domains-hosts/brute_hosts                    | 1.0     | installed | 2019-06-24 |   |   |
  | recon/domains-hosts/builtwith                      | 1.1     | installed | 2021-08-24 |   | * |
  | recon/domains-hosts/censys_domain                  | 2.0     | disabled  | 2021-05-10 | * | * |
  | recon/domains-hosts/certificate_transparency       | 1.2     | installed | 2019-09-16 |   |   |
  | recon/domains-hosts/google_site_web                | 1.0     | installed | 2019-06-24 |   |   |
  | recon/domains-hosts/hackertarget                   | 1.1     | installed | 2020-05-17 |   |   |
  | recon/domains-hosts/mx_spf_ip                      | 1.0     | installed | 2019-06-24 |   |   |
  | recon/domains-hosts/netcraft                       | 1.1     | installed | 2020-02-05 |   |   |
  | recon/domains-hosts/shodan_hostname                | 1.1     | installed | 2020-07-01 | * | * |
  | recon/domains-hosts/spyse_subdomains               | 1.1     | installed | 2021-08-24 |   | * |
  | recon/domains-hosts/ssl_san                        | 1.0     | installed | 2019-06-24 |   |   |
  | recon/domains-hosts/threatcrowd                    | 1.0     | installed | 2019-06-24 |   |   |
  | recon/domains-hosts/threatminer                    | 1.0     | installed | 2019-06-24 |   |   |
  | recon/domains-vulnerabilities/ghdb                 | 1.1     | installed | 2019-06-26 |   |   |
  | recon/domains-vulnerabilities/xssed                | 1.1     | installed | 2020-10-18 |   |   |
  | recon/hosts-domains/migrate_hosts                  | 1.1     | installed | 2020-05-17 |   |   |
  | recon/hosts-hosts/censys_query                     | 2.0     | disabled  | 2021-05-10 | * | * |
  | recon/hosts-hosts/virustotal                       | 1.0     | installed | 2019-06-24 |   | * |
  | recon/netblocks-hosts/virustotal                   | 1.0     | installed | 2019-06-24 |   | * |
  +-----------------------------------------------------------------------------------------------+

  D = Has dependencies. See info for details.
  K = Requires keys. See info for details.

[recon-ng][test] >

We can see that some modules require API keys and/or additional dependencies to be used. Running marketplace info on a module path describes the module:

[recon-ng][test] > marketplace info recon/domains-hosts/google_site_web  

  +---------------------------------------------------------------------------------------------------------------------------------+
  | path          | recon/domains-hosts/google_site_web                                                                             |
  | name          | Google Hostname Enumerator                                                                                      |
  | author        | Tim Tomes (@lanmaster53)                                                                                        |
  | version       | 1.0                                                                                                             |
  | last_updated  | 2019-06-24                                                                                                      |
  | description   | Harvests hosts from Google.com by using the 'site' search operator. Updates the 'hosts' table with the results. |
  | required_keys | []                                                                                                              |
  | dependencies  | []                                                                                                              |
  | files         | []                                                                                                              |
  | status        | installed                                                                                                       |
  +---------------------------------------------------------------------------------------------------------------------------------+

Let us try using this module for subdomain enumerator given a seed domain. Running modules load command with the module path activates the modules and gives us a subshell for running commands within context of the module.

[recon-ng][test] > modules load recon/domains-hosts/google_site_web 
[recon-ng][test][google_site_web] >

For example, running info gives us a simple description on the current module and it’s parameters.

recon-ng][test][google_site_web] > info

      Name: Google Hostname Enumerator
    Author: Tim Tomes (@lanmaster53)
   Version: 1.0

Description:
  Harvests hosts from Google.com by using the 'site' search operator. Updates the 'hosts' table with
  the results.

Options:
  Name    Current Value  Required  Description
  ------  -------------  --------  -----------
  SOURCE  default        yes       source of input (see 'info' for details)

Source Options:
  default        SELECT DISTINCT domain FROM domains WHERE domain IS NOT NULL
  <string>       string representing a single input
  <path>         path to a file containing a list of inputs
  query <sql>    database query returning one column of inputs

Let us set the SOURCE option to a domain of a company that has an open bug bounty program and run the module.

[recon-ng][test][google_site_web] > options set SOURCE tencent.com
SOURCE => tencent.com
[recon-ng][test][google_site_web] > run

-----------
TENCENT.COM
-----------
[*] Searching Google for: site:tencent.com
[*] Country: None
[*] Host: buy.cloud.tencent.com
[*] Ip_Address: None
[*] Latitude: None
[*] Longitude: None
[*] Notes: None
[*] Region: None
[*] --------------------------------------------------
[*] Country: None
[*] Host: intl.cloud.tencent.com
[*] Ip_Address: None
[*] Latitude: None
[*] Longitude: None
[*] Notes: None
[*] Region: None

...
[*] Searching Google for: site:tencent.com -site:buy.cloud.tencent.com -site:intl.cloud.tencent.com -site:meeting.tencent.com -site:open.tencent.com -site:tdesign.tencent.com -site:s.tencent.com -site:dnspod.cloud.tencent.com -site:www.tencent.com -site:ioa.tencent.com -site:market.cloud.tencent.com -site:ad.tencent.com -site:xiaowei.tencent.com -site:spd.tencent.com -site:opensource.tencent.com -site:isux.tencent.com -site:hiflow.tencent.com -site:tmc.tencent.com -site:ur.tencent.com -site:partner.cloud.tencent.com -site:ieg.tencent.com -site:app.cloud.tencent.com -site:8000.tencent.com -site:drug.ai.tencent.com -site:code.tencent.com -site:cloud.tencent.com -site:mxd.tencent.com -site:tea-lab.tencent.com -site:retail.tencent.com -site:careers.tencent.com -site:gss.tencent.com -site:todo.tencent.com -site:gameinstitute.tencent.com -site:broker.tencent.com -site:tapd.tencent.com -site:light.mofyi.tencent.com -site:tdw.tencent.com -site:aiarena.tencent.com -site:developers.tencent.com -site:om.tencent.com -site:keenlab.tencent.com -site:gwb.tencent.com -site:service.security.tencent.com -site:cdc.tencent.com -site:mtp.tencent.com -site:calendar.tencent.com -site:design.tencent.com -site:ess.tencent.com -site:gcloud.tencent.com -site:dsic.tencent.com -site:tengshi.tencent.com -site:matrix.tencent.com -site:research.tencent.com -site:jarvislab.tencent.com -site:x5.tencent.com -site:rtx.tencent.com -site:aidata.tencent.com -site:fapiao.tencent.com -site:beacon.tencent.com -site:edu.tencent.com -site:tcaplusdb.tencent.com -site:cii.tencent.com -site:healthcare.tencent.com -site:wemake.tencent.com -site:ai.tencent.com -site:security.tencent.com -site:arc.tencent.com -site:idc.tencent.com -site:quantum.tencent.com -site:tosp.tencent.com -site:en.security.tencent.com -site:app.v.tencent.com -site:blade.tencent.com -site:ipr.tencent.com -site:multimedia.tencent.com -site:jubao.tencent.com -site:xlab.tencent.com
[*] Country: None
[*] Host: tengyun.tencent.com
[*] Ip_Address: None
[*] Latitude: None
[*] Longitude: None
[*] Notes: None
[*] Region: None
[*] --------------------------------------------------
[*] Country: None
[*] Host: supplier.tencent.com
[*] Ip_Address: None
[*] Latitude: None
[*] Longitude: None
[*] Notes: None
[*] Region: None
[*] --------------------------------------------------
[*] Country: None
[*] Host: yzt.tencent.com
[*] Ip_Address: None
[*] Latitude: None
[*] Longitude: None
[*] Notes: None
[*] Region: None

...

[*] Searching Google for: site:tencent.com -site:buy.cloud.tencent.com -site:intl.cloud.tencent.com -site:meeting.tencent.com -site:open.tencent.com -site:tdesign.tencent.com -site:s.tencent.com -site:dnspod.cloud.tencent.com -site:www.tencent.com -site:ioa.tencent.com -site:market.cloud.tencent.com -site:ad.tencent.com -site:xiaowei.tencent.com -site:spd.tencent.com -site:opensource.tencent.com -site:isux.tencent.com -site:hiflow.tencent.com -site:tmc.tencent.com -site:ur.tencent.com -site:partner.cloud.tencent.com -site:ieg.tencent.com -site:app.cloud.tencent.com -site:8000.tencent.com -site:drug.ai.tencent.com -site:code.tencent.com -site:cloud.tencent.com -site:mxd.tencent.com -site:tea-lab.tencent.com -site:retail.tencent.com -site:careers.tencent.com -site:gss.tencent.com -site:todo.tencent.com -site:gameinstitute.tencent.com -site:broker.tencent.com -site:tapd.tencent.com -site:light.mofyi.tencent.com -site:tdw.tencent.com -site:aiarena.tencent.com -site:developers.tencent.com -site:om.tencent.com -site:keenlab.tencent.com -site:gwb.tencent.com -site:service.security.tencent.com -site:cdc.tencent.com -site:mtp.tencent.com -site:calendar.tencent.com -site:design.tencent.com -site:ess.tencent.com -site:gcloud.tencent.com -site:dsic.tencent.com -site:tengshi.tencent.com -site:matrix.tencent.com -site:research.tencent.com -site:jarvislab.tencent.com -site:x5.tencent.com -site:rtx.tencent.com -site:aidata.tencent.com -site:fapiao.tencent.com -site:beacon.tencent.com -site:edu.tencent.com -site:tcaplusdb.tencent.com -site:cii.tencent.com -site:healthcare.tencent.com -site:wemake.tencent.com -site:ai.tencent.com -site:security.tencent.com -site:arc.tencent.com -site:idc.tencent.com -site:quantum.tencent.com -site:tosp.tencent.com -site:en.security.tencent.com -site:app.v.tencent.com -site:blade.tencent.com -site:ipr.tencent.com -site:multimedia.tencent.com -site:jubao.tencent.com -site:xlab.tencent.com -site:tengyun.tencent.com -site:supplier.tencent.com -site:yzt.tencent.com -site:rfq.tencent.com -site:moa.tencent.com -site:growing.tencent.com -site:we.tencent.com -site:youshu.tencent.com -site:cloudsec.tencent.com -site:yunding.tencent.com -site:dmp.tencent.com -site:privacy.tencent.com -site:rg.tencent.com -site:agency.tencent.com -site:tpa.tencent.com -site:tssp.tencent.com -site:hebao.tencent.com -site:yixing.tencent.com -site:sign2.tencent.com -site:tfdp.tencent.com
[!] Google CAPTCHA triggered. No bypass available.

-------
SUMMARY
-------
[*] 96 total (96 new) hosts found.

We see that the modules incrementally discovers new subdomains by using site: operator to get Google search results matching the seed domain, but also using minus sign to remove the already known subdomains from search result pages. It was able to scrape 96 subdomains before running into captcha that Google gives the user to solve when it detects too many search requests too quickly from single IP address. To see all the data it got so far we can run show hosts.

At this point we have a bunch of subdomains, but no IP addresses related to them. We can enrich the data in the hosts table by loading and running another module that will perform DNS requests for the subdomains:

[recon-ng][test][google_site_web] > modules load recon/hosts-hosts/resolve
[recon-ng][test][resolve] > info

      Name: Hostname Resolver
    Author: Tim Tomes (@lanmaster53)
   Version: 1.0

Description:
  Resolves the IP address for a host. Updates the 'hosts' table with the results.

Options:
  Name    Current Value  Required  Description
  ------  -------------  --------  -----------
  SOURCE  default        yes       source of input (see 'info' for details)

Source Options:
  default        SELECT DISTINCT host FROM hosts WHERE host IS NOT NULL AND ip_address IS NULL
  <string>       string representing a single input
  <path>         path to a file containing a list of inputs
  query <sql>    database query returning one column of inputs

Comments:
  * Note: Nameserver must be in IP form.

[recon-ng][test][resolve] > run
[*] buy.cloud.tencent.com => 158.79.1.174
[*] buy.cloud.tencent.com => 158.79.1.160
[*] intl.cloud.tencent.com => 158.79.1.160
[*] intl.cloud.tencent.com => 158.79.1.174
[*] meeting.tencent.com => 203.205.137.236
[*] open.tencent.com => 129.226.107.102
[*] open.tencent.com => 129.226.106.5
[*] tdesign.tencent.com => 158.79.1.56
[*] tdesign.tencent.com => 158.79.1.178
[*] tdesign.tencent.com => 158.79.1.187
[*] tdesign.tencent.com => 158.79.1.185
[*] s.tencent.com => 43.135.106.184
[*] s.tencent.com => 43.135.106.117
[*] dnspod.cloud.tencent.com => 42.194.253.127
[*] www.tencent.com => 43.152.54.219
[*] www.tencent.com => 43.152.56.217
[*] ioa.tencent.com => 119.28.121.91
[*] market.cloud.tencent.com => 183.2.144.115
...

Running show hosts again will show us that ip_address column is now filled. Internally each workspace will have SQLite database for persisting the data and the recon-ng command line gives us a way to run SQL queries on it:

[recon-ng][test][resolve] > db
Interfaces with the workspace's database

Usage: db <delete|insert|notes|query|schema> [...]

[recon-ng][test][resolve] > db query SELECT * FROM hosts LIMIT 10;

  +----------------------------------------------------------------------------------------------------------------+
  |           host           |    ip_address   | region | country | latitude | longitude | notes |      module     |
  +----------------------------------------------------------------------------------------------------------------+
  | buy.cloud.tencent.com    | 158.79.1.174    |        |         |          |           |       | google_site_web |
  | intl.cloud.tencent.com   | 158.79.1.160    |        |         |          |           |       | google_site_web |
  | meeting.tencent.com      | 203.205.137.236 |        |         |          |           |       | google_site_web |
  | open.tencent.com         | 129.226.107.102 |        |         |          |           |       | google_site_web |
  | tdesign.tencent.com      | 158.79.1.56     |        |         |          |           |       | google_site_web |
  | s.tencent.com            | 43.135.106.184  |        |         |          |           |       | google_site_web |
  | dnspod.cloud.tencent.com | 42.194.253.127  |        |         |          |           |       | google_site_web |
  | www.tencent.com          | 43.152.54.219   |        |         |          |           |       | google_site_web |
  | ioa.tencent.com          | 119.28.121.91   |        |         |          |           |       | google_site_web |
  | market.cloud.tencent.com | 183.2.144.115   |        |         |          |           |       | google_site_web |
  +----------------------------------------------------------------------------------------------------------------+

[*] 10 rows returned

For further analysis of data outside recon-ng command line we can take a snapshot of the database:

[recon-ng][test][resolve] > back
[recon-ng][test] > snapshots take
[*] Snapshot created: snapshot_20220906192805.db

The newly created file at ~/.recon-ng/workspaces/test/ contains the exported SQLite database that can be further analysed within Jupyter notebook or some other environment. Note that we used the back command to exit the module sub-shell since some commands are not available there.

Now let us generate an HTML report. There’s also a module for that.

[recon-ng][test] > modules load reporting/html
[recon-ng][test][html] > info

      Name: HTML Report Generator
    Author: Tim Tomes (@lanmaster53)
   Version: 1.0

Description:
  Creates an HTML report.

Options:
  Name      Current Value                                     Required  Description
  --------  -------------                                     --------  -----------
  CREATOR                                                     yes       use creator name in the report footer
  CUSTOMER                                                    yes       use customer name in the report header
  FILENAME  /Users/rl/.recon-ng/workspaces/test/results.html  yes       path and filename for report output
  SANITIZE  True                                              yes       mask sensitive data in the report

[recon-ng][test][html] > options set CREATOR rl1987
CREATOR => rl1987
[recon-ng][test][html] > options set CUSTOMER Shady Industries, Inc
CUSTOMER => Shady Industries, Inc
[recon-ng][test][html] > run
[*] Report generated at '/Users/rl/.recon-ng/workspaces/test/results.html'.

Screenshot

But what if we wanted to have a script to run everything we just did now in a reproducible manner? Running recon-ng as a subprocess within some programming language and feeding the commands through standard input could work, but is rather clunky. The good news is that we can create a text file with one command per line and use -r CLI option to run all the commands in batch mode (don’t forget the exit command at the end to make it finish without human intervention):

$ recon-ng -r script

cURL beyond the basics

Sun, 04 Sep 2022 00:00:00 +0000

Curl is a prominent CLI tool and library for performing client-side requests in a number of application layer protocols. Commonly it is used for HTTP requests. However, there is more to curl than that. We will go through some less known features of curl that can be used in web scraping and automation systems.

Debugging

Sometimes things fail and require us to take a deeper look into technical details of what is happening. Running curl with --verbose (or just -v) gives us a certain level of technical details regarding connection establishment and protocol messages being transmitted:

$ curl http://httpbin.org/status/401 -v 
*   Trying 34.227.213.82:80...
* Connected to httpbin.org (34.227.213.82) port 80 (#0)
> GET /status/401 HTTP/1.1
> Host: httpbin.org
> User-Agent: curl/7.79.1
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 401 UNAUTHORIZED
< Date: Sat, 03 Sep 2022 07:28:47 GMT
< Server: gunicorn/19.9.0
< WWW-Authenticate: Basic realm="Fake Realm"
< Access-Control-Allow-Origin: *
< Access-Control-Allow-Credentials: true
< Cache-Control: proxy-revalidate
< Content-Length: 0
< Connection: Keep-Alive
< 
* Connection #0 to host httpbin.org left intact

However, we may want to look deeper. There are CLI options that gives us even more detailed output for troubleshooting. --trace saves full dump of data being sent and received into a text file (or standard output if we use - instead of file path):

$ curl http://httpbin.org/status/401 --trace -
== Info:   Trying 34.227.213.82:80...
== Info: Connected to httpbin.org (34.227.213.82) port 80 (#0)
=> Send header, 85 bytes (0x55)
0000: 47 45 54 20 2f 73 74 61 74 75 73 2f 34 30 31 20 GET /status/401 
0010: 48 54 54 50 2f 31 2e 31 0d 0a 48 6f 73 74 3a 20 HTTP/1.1..Host: 
0020: 68 74 74 70 62 69 6e 2e 6f 72 67 0d 0a 55 73 65 httpbin.org..Use
0030: 72 2d 41 67 65 6e 74 3a 20 63 75 72 6c 2f 37 2e r-Agent: curl/7.
0040: 37 39 2e 31 0d 0a 41 63 63 65 70 74 3a 20 2a 2f 79.1..Accept: */
0050: 2a 0d 0a 0d 0a                                  *....
== Info: Mark bundle as not supporting multiuse
<= Recv header, 27 bytes (0x1b)
0000: 48 54 54 50 2f 31 2e 31 20 34 30 31 20 55 4e 41 HTTP/1.1 401 UNA
0010: 55 54 48 4f 52 49 5a 45 44 0d 0a                UTHORIZED..
<= Recv header, 37 bytes (0x25)
0000: 44 61 74 65 3a 20 53 61 74 2c 20 30 33 20 53 65 Date: Sat, 03 Se
0010: 70 20 32 30 32 32 20 30 37 3a 33 36 3a 32 35 20 p 2022 07:36:25 
0020: 47 4d 54 0d 0a                                  GMT..
<= Recv header, 25 bytes (0x19)
0000: 53 65 72 76 65 72 3a 20 67 75 6e 69 63 6f 72 6e Server: gunicorn
0010: 2f 31 39 2e 39 2e 30 0d 0a                      /19.9.0..
<= Recv header, 44 bytes (0x2c)
0000: 57 57 57 2d 41 75 74 68 65 6e 74 69 63 61 74 65 WWW-Authenticate
0010: 3a 20 42 61 73 69 63 20 72 65 61 6c 6d 3d 22 46 : Basic realm="F
0020: 61 6b 65 20 52 65 61 6c 6d 22 0d 0a             ake Realm"..
<= Recv header, 32 bytes (0x20)
0000: 41 63 63 65 73 73 2d 43 6f 6e 74 72 6f 6c 2d 41 Access-Control-A
0010: 6c 6c 6f 77 2d 4f 72 69 67 69 6e 3a 20 2a 0d 0a llow-Origin: *..
<= Recv header, 40 bytes (0x28)
0000: 41 63 63 65 73 73 2d 43 6f 6e 74 72 6f 6c 2d 41 Access-Control-A
0010: 6c 6c 6f 77 2d 43 72 65 64 65 6e 74 69 61 6c 73 llow-Credentials
0020: 3a 20 74 72 75 65 0d 0a                         : true..
<= Recv header, 33 bytes (0x21)
0000: 43 61 63 68 65 2d 43 6f 6e 74 72 6f 6c 3a 20 70 Cache-Control: p
0010: 72 6f 78 79 2d 72 65 76 61 6c 69 64 61 74 65 0d roxy-revalidate.
0020: 0a                                              .
<= Recv header, 19 bytes (0x13)
0000: 43 6f 6e 74 65 6e 74 2d 4c 65 6e 67 74 68 3a 20 Content-Length: 
0010: 30 0d 0a                                        0..
<= Recv header, 24 bytes (0x18)
0000: 43 6f 6e 6e 65 63 74 69 6f 6e 3a 20 4b 65 65 70 Connection: Keep
0010: 2d 41 6c 69 76 65 0d 0a                         -Alive..
<= Recv header, 2 bytes (0x2)
0000: 0d 0a                                           ..
== Info: Connection #0 to host httpbin.org left intact

If we wanted to skip hexdumps of protocol messages we can use --trace-ascii instead of --trace. To augment the debug output with timing information we can use --trace-time:

$ curl http://httpbin.org/status/401 --trace-ascii - --trace-time
14:38:54.540384 == Info:   Trying 54.147.68.244:80...
14:38:54.573724 == Info: Connected to httpbin.org (54.147.68.244) port 80 (#0)
14:38:54.573928 => Send header, 85 bytes (0x55)
0000: GET /status/401 HTTP/1.1
001a: Host: httpbin.org
002d: User-Agent: curl/7.79.1
0046: Accept: */*
0053: 
14:38:55.131822 == Info: Mark bundle as not supporting multiuse
14:38:55.131875 <= Recv header, 27 bytes (0x1b)
0000: HTTP/1.1 401 UNAUTHORIZED
14:38:55.131907 <= Recv header, 37 bytes (0x25)
0000: Date: Sat, 03 Sep 2022 07:38:55 GMT
14:38:55.131932 <= Recv header, 25 bytes (0x19)
0000: Server: gunicorn/19.9.0
14:38:55.131958 <= Recv header, 44 bytes (0x2c)
0000: WWW-Authenticate: Basic realm="Fake Realm"
14:38:55.131983 <= Recv header, 32 bytes (0x20)
0000: Access-Control-Allow-Origin: *
14:38:55.132004 <= Recv header, 40 bytes (0x28)
0000: Access-Control-Allow-Credentials: true
14:38:55.132028 <= Recv header, 33 bytes (0x21)
0000: Cache-Control: proxy-revalidate
14:38:55.132050 <= Recv header, 19 bytes (0x13)
0000: Content-Length: 0
14:38:55.132070 <= Recv header, 24 bytes (0x18)
0000: Connection: Keep-Alive
14:38:55.132091 <= Recv header, 2 bytes (0x2)
0000: 
14:38:55.132122 == Info: Connection #0 to host httpbin.org left intact

This can be used with --trace, --trace-ascii and --verbose.

Dictionary lookups with DICT protocol

DICT (RFC 2229) is a application level protocol for performing dictionary lookups. It can be used via curl like this:

$ curl dict://dict.org/d:linux       
220 dict.dict.org dictd 1.12.1/rf on Linux 4.19.0-10-amd64 <auth.mime> <138373555.5356.1662191318@dict.dict.org>
250 ok
150 1 definitions retrieved
151 "linux" wn "WordNet (r) 3.0 (2006)"
Linux
    n 1: an open-source version of the UNIX operating system
.
250 ok [d/m/c = 1/0/30; 0.000r 0.000u 0.000s]
221 bye [d/m/c = 0/0/0; 0.000r 0.000u 0.000s]

This may seem like it’s nothing to write home about. After all, we can search Google with define: and the word we want to lookup. Scraping Google SERPs either directly or through some SaaS API is not hard. However this provides an example of a fairly obscure network protocol that curl has support for.

Transferring files via (S)FTP

Let us something bit more useful now. Curl can be used to transfer files via FTP, SFTP and FTPS protocols. For example, we can use curl to list files on FTP server and download them:

$ curl --list-only ftp://ftp.sunet.se             
HEADER.html
Public
about
cdimage
conspiracy
debian
debian-cd
favicon.ico
images
mirror
pub
releases
robots.txt
tails
ubuntu
$ curl --list-only ftp://ftp.sunet.se/debian/
README
README.CD-manufacture
README.html
README.mirrors.html
README.mirrors.txt
dists
doc
extrafiles
indices
ls-lR.gz
pool
project
tools
zzz-dists
$ curl ftp://ftp.sunet.se/debian/README -o README

If we wanted to upload a local file to remote path at FTP server we could use -T or --upload-file.

However plain text anonymous servers are going out of fashion within last couple decades. When we have SSH access to a remote server we can also use SFTP - a more modern file transfer protocol that is built upon the foundation of SSH. This is not merely FTP over SSH, but a separate protocol designed from ground up by IETF. It is also more generalised than the legacy SCP protocol.

To use SFTP with curl, we would use the same URL format, except we would have sftp:// at the beginning. If username/password auth is required we can pass the credentials in colon-separated form via --user option.

If we are trying to connect to FTPS (FTP over TLS) server we would use ftps:// for the URLs.

Sending and receiving email

Curl also supports SMTP protocol for email sending. Let us try sending an email via Sendgrid. First we need to prepare a text file with some email headers and content. This is what we save into email.txt:

From: No Reply <noreply@keyspace.lt>
To: Test User <test2048@mailinator.com>
Subject: Test email

A specter is haunting the modern world, the specter of crypto anarchy.

A command to send this email is the following:

$ curl --insecure --ssl smtp://smtp.sendgrid.com:587 --user "apikey:[REDACTED]" --mail-rcpt "test2048@mailinator.com" --mail-from "noreply@keyspace.lt" --upload-file email.txt

The redacted part is Sendgrid API key that we use as SMTP password.

Curl can be used to send emails, but what about receiving them? This is also possible via POP3 and IMAP protocols.

Let us first list message IDs and sizes via POP3 protocol:

$ curl pop3://pop3.rambler.ru --user "ermakova_g01bm@rambler.ru:[REDACTED]"
1 4388

We have a single message with ID 1 and size 4388. It can be downloaded by using the message ID as URL path:

$ curl pop3://pop3.rambler.ru/1 --user "ermakova_g01bm@rambler.ru:[REDACTED]"

Things are a bit different with IMAP, as IMAP protocol supports different mailboxes that we can list:

$ curl --ssl imap://imap.rambler.ru/ --user "ermakova_g01bm@rambler.ru:[REDACTED]"    
* LIST (\HasNoChildren \UnMarked \Sent) "/" SentBox
* LIST (\HasNoChildren \Junk) "/" Spam
* LIST (\HasNoChildren \Marked \Trash) "/" Trash
* LIST (\HasNoChildren \Drafts) "/" DraftBox
* LIST (\HasNoChildren) "/" INBOX

To check the status of mailbox, let us send examine IMAP command:

$ curl --ssl "imap://mail.rambler.ru/" -X "examine INBOX" --user "ermakova_g01bm@rambler.ru:[REDACTED]" 
* FLAGS (\Answered \Flagged \Deleted \Seen \Draft)
* OK [PERMANENTFLAGS ()] Read-only mailbox.
* 2 EXISTS
* 0 RECENT
* OK [UIDVALIDITY 1644072147] UIDs valid
* OK [UIDNEXT 20] Predicted next UID
* OK [HIGHESTMODSEQ 37] Highest

To get a single message at specific index, query it like this:

$ curl --ssl "imap://mail.rambler.ru/INBOX;MAILINDEX=1" --user "ermakova_g01bm@rambler.ru:[REDACTED]"

libcurl and it’s bindings

Curl is not only a CLI tool, but also a C library. If we have a curl snippet we can add --libcurl with file path to generate some boilerplate C code:

$ curl http://httpbin.org/status/401 --libcurl example.c

It will generate something like this:

/********* Sample code generated by the curl command line tool **********
 * All curl_easy_setopt() options are documented at:
 * https://curl.se/libcurl/c/curl_easy_setopt.html
 ************************************************************************/
#include <curl/curl.h>

int main(int argc, char *argv[])
{
  CURLcode ret;
  CURL *hnd;

  hnd = curl_easy_init();
  curl_easy_setopt(hnd, CURLOPT_BUFFERSIZE, 102400L);
  curl_easy_setopt(hnd, CURLOPT_URL, "http://httpbin.org/status/401");
  curl_easy_setopt(hnd, CURLOPT_NOPROGRESS, 1L);
  curl_easy_setopt(hnd, CURLOPT_USERAGENT, "curl/7.85.0");
  curl_easy_setopt(hnd, CURLOPT_MAXREDIRS, 50L);
  curl_easy_setopt(hnd, CURLOPT_HTTP_VERSION, (long)CURL_HTTP_VERSION_2TLS);
  curl_easy_setopt(hnd, CURLOPT_FTP_SKIP_PASV_IP, 1L);
  curl_easy_setopt(hnd, CURLOPT_TCP_KEEPALIVE, 1L);

  /* Here is a list of options the curl code used that cannot get generated
     as source easily. You may choose to either not use them or implement
     them yourself.

  CURLOPT_WRITEDATA set to a objectpointer
  CURLOPT_INTERLEAVEDATA set to a objectpointer
  CURLOPT_WRITEFUNCTION set to a functionpointer
  CURLOPT_READDATA set to a objectpointer
  CURLOPT_READFUNCTION set to a functionpointer
  CURLOPT_SEEKDATA set to a objectpointer
  CURLOPT_SEEKFUNCTION set to a functionpointer
  CURLOPT_ERRORBUFFER set to a objectpointer
  CURLOPT_STDERR set to a objectpointer
  CURLOPT_HEADERFUNCTION set to a functionpointer
  CURLOPT_HEADERDATA set to a objectpointer

  */

  ret = curl_easy_perform(hnd);

  curl_easy_cleanup(hnd);
  hnd = NULL;

  return (int)ret;
}
/**** End of sample code ****/

If you are not working in fairly deep systems programming you may not be willing to code in C. However, there’s libcurl bindings available in a lot of languages:

C++ - curlpp
Go - go-curl
Haskell - curl Haskell package
Java - curl-java
JavaScript - node-libcurl
PHP - PHP curl library
Python - PyCURL
Rust - curl-rust

If you merely need to do some basic HTTP requests using libcurl is likely to be an overkill. For example, Python requests module is far easier to use than PyCURL. However, if you’re doing some nonstandard stuff libcurl might be a good abstraction layer to base your code upon. For example, once I had to integrate proxy servers that require TLS handshake to be performed before HTTP CONNECT message is sent by the client. Using such proxy servers is possible with PyCURL.

Posts on Trickster Dev

Advanced network traffic interception techniques with mitmproxy

Reverse proxy mode

Transparent proxy mode

mitmproxy as Wireguard VPN server

SOCKS proxy mode

mitmproxy as DNS server

Passing traffic to upstream proxy

mitmproxy virtual network interface

Scraping Next.js web sites in 2025

Setting up rooted Android emulator with Frida and mitmproxy

jq: tool and language for JSON processing

Katana: web crawler for offensive security and web exploration

Lesser known parts of Python standard library

Simple ways to find exposed sensitive information

Understanding SOCKS protocol

Scraping Clutch for B2B company data

SSH tips and tricks

SSH escape sequences

Single command through SSH

2FA on SSH login

SSH tunneling with -L

SSH connection as local SOCKS proxy

SSH server behind NAT via Tor Onion Service

Mounting directory subtree via sshfs

Using SSH programmatically via Paramiko/Fabric

Retrospective scraping of historical data from Internet Archive snapshots

Obscure corners of a standard macOS installation

Apache httpd

say(1) and TTS in Edit menu

screen(1)

Grapher

Digital Colour Meter

Font Book

/usr/share/calendar and calendar(1)

pbcopy(1) and pbpaste(1)

networkQuality

Scraping data from Google Places API

Extracting data from elasticsearch API

Setting up mitmproxy with iOS 17.1

Parsing HTML documents with Beautiful Soup

Solving a simple JS challenge with sandboxing

Abusing GOAT mobile API for data scraping

Scrapy by example: scraping company data from Yelp

Back to the terminal: the new era of CLI and TUI software

Carbonyl: 21st Century Lynx

ag, ripgrep - the new grep(1)

bottom, btop, glances - better kind of top(1)

Terminal Image Viewer (tiv)

HTTPie - command line API client

jq - CLI tool and DSL for JSON processing

tshark - CLI equivalent of Wireshark

TermShark

mitmproxy

neovim

GitHub CLI tool

mapscii

Visidata

Radare2

Rich and Textual

Restringer: modular JavaScript deobfuscator

Using Scrapy-playwright for integrating browser automation with Scrapy

Self-defending JS code and debugger traps

Abusing Ahrefs Backlink Checker

Don't JSFuck with me: Part 3

Prompting the LLM is no substitute for knowledge work

Understanding JavaScript packers

Scraping product data from Nike.com

How do they (try to) block bots: a broad review of automation countermeasures

IP level

Geofencing

Rate limiting

Filtering by AS/ISP

Filtering by IP reputation

HTTPS level

HTTP headers and cookies

HTTP/2 fingerprinting

TLS fingerprinting

Application level

JS environment checking

SSH tunneling with `-L`