wiki:I2P_Browser_develop_n_hacks

Version 26 (modified by Meeh, 3 weeks ago) (diff)

I2P Browser Development And Hacking

Howto get started with the build system

Howto get started with firefox

The firefox codebase is no doubt massive, for an overview, take a look at Directory Overview.

It's worth having a look at Tor's trac wiki as well, as they also have browser hacking pages - but with more or less no duplicity of info between here and there.

As it is widely used in Mozilla, it makes sense to explain how JavaScript? and C++ relate to each other in the Mozilla source code. C++ is a compiled language, while JavaScript? is an interpreted language. JavaScript? is most commonly known as a technology used to implement web sites. However, the developers of Mozilla decided that the Mozilla source code itself should consist of a mixture of both languages.

When you start the application, the C/C++ components start first. But in an early stage, a technology called XPConnect gets initialized that enables the use of interpreted JavaScript? at runtime. In fact, a Mozilla browser distribution consists of both compiled C++ and uncompiled JS files.

Also note, when surfing web pages which use JavaScript?, that JavaScript? code is executed within a sand box, and does not have access to Mozilla’s internal objects. Only those objects that are exposed by DOM (Document Object Model) are accessible.

Whenever JS is mentioned in this document, it is meant as a technology to contribute to internal Mozilla functionality in most cases unless otherwise spesified. JavaScript? is mostly used in those areas of the source code that care for user interface events. Most of the following document will explain the C++ aspect of the application.

The firefox interfaces concept

The concept of interfaces is something that is used in the CORBA technology, for example. Both CORBA and Mozilla use a similar language to describe interfaces, which is XPIDL (IDL means Interface Definition Language).

In a CORBA environment, life is more restrictive and difficult, because you have inter-process and inter-network communication – something which Mozilla is not actively using. In a distributed CORBA environment, it is difficult to change the components of an interface, because you are usually unable to replace all running systems at the same time. If you want to change something, you have to define a new version of an interface, but you might still be required to support the old one.

As Mozilla is not a distributed application as of writing, it is currently possible to change most interfaces as the development process requires it. But because the Mozilla browser runs embedded in some environments, those environments must be able to rely on a fixed interface; therefore, interfaces can be frozen. This state is usually indicated in the interface definition. As Mozilla stabilizes over time, the ratio of frozen to not frozen interfaces is likely to increase.

One step of building Mozilla is automatically translating the interface definition files into C/C++ header files. That’s the job of Mozilla’s own IDL compiler, xpidl.

Besides of the methods and data members, interfaces have additional attributes. They have a UUID, a number to uniquely identify an interface. Interfaces can have the scriptable attribute, which means they will be accessible from the JavaScript? code. A scriptable interface is restricted to only use data types for parameters that are valid within the JavaScript? runtime.

XPCOM / nsISupports / nsCOMPtr

XPCOM is Mozilla’s own implementation of COM – the component object model. The XP in front means it is cross-platform (do not confuse this with XP as it appears in product names for a certain operating system manufacturer). The fact that it is cross-platform makes XPCOM more versatile than any other form of COM.

You should definitely read the introductory documents on XPCOM on mozilla.org. To get you started, one could say that XPCOM is the engine that makes the COM concept work. This includes playing the role of an object broker.

Typically, an interface describes an object that can be used to get a job done. If you have a job to do, you need to request an implementation that provides the interface. That implementation can reside within another component. To decide which particular implementation you want, you are using a contract ID, which is a text based identifier. The contract ID is a contract on the behaviour of the implementation, accessible using the well defined interface. The XPCOM runtime system knows which class implements which contract ID, and which component provides it.

Even if your code stays completely within one component, and therefore using COM is not a strict requirement, it is very often used anyway. One reason is flexibility. Another is to allow sharing functionality with those portions of the Mozilla logic that are implemented using JavaScript?. Mozilla provides a technology called XPConnect that enables communication between interpreted JavaScript? and compiled C++ at runtime.

Whenever you request an instance of a COM object at runtime, a class object will be created and you will be given an interface pointer. For several reasons it was decided that these instances are reference counted. One reason is efficiency, since making unnecessary copies of objects should be avoided. Another requirement is that when data objects must be passed between threads, each thread needs to keep a pointer to the same data object in memory. Finally, the same data object might be referenced by multiple components, or stored in multiple lists.

As the lifetime of each reference is different, it is easiest to have each object maintain a reference count, to remember how often it is currently referenced by something else. When you are given a reference to an object (be it from the XPCOM engine directly or from a function call), you have to make sure that you care for reference counting. You must tell the object whether you want to keep a reference to it, or whether you are finished with it, and remove the reference. That way, the object can decide on its own whether it is still needed. When its not needed anymore, it deletes itself from memory.

In order to implement this generic functionality, all classes in Mozilla that implement any interface share the common base class nsISupports, which implements the reference counter and automatic destruction functionality. A similar base class exists in every implementation of COM.

There is a general rule that you must clean up what you allocate. For instance, if you add a reference, you are strongly encouraged to release the reference as soon as it is no longer needed. If you don’t, you might cause problems such as memory leaks.

In C++, this can be done by explicit method calls to methods of the nsISupports base class. But calling these methods is not only easy to forget, but it also makes your code less readable – especially since many functions/methods have multiple exit points (i.e. return statements).

You have to make sure that you release all your referenced objects in each of your exit points. To make this easier, and to not have to repeat many release calls, a general helper class has been provided for dealing with pointers to COM objects, whose name is nsCOMPtr. This is something special to XPCOM and makes coding COM much easier. It simulates a pointer by overriding certain operators. Although there might be some edge cases, the following general rule should be followed for nearly all code: Whenever you’d expect to use a pointer variable “interface*” to an object that implements an interface, use a local “nsCOMPtr<interface>” variable instead. As soon as this pointer goes “out of scope”, its destructor will automatically decrease the reference count, triggering destruction if possible.

In interpreted JavaScript? this is easier to code, because of garbage collection. There is some magic that automatically releases the references when possible. However, this magic requires that you don’t introduce cycles. For example, if you have two objects, and each contain a reference to the other, but nobody else keeps a reference to them, this can not be detected. Those objects will live in memory for the rest of program execution.

Exceptions / nsresult

Code execution can fail at runtime. One programming mechanism to deal with failure is to use exceptions. While Mozilla uses Exceptions in its JavaScript? code portions, it does not in C++. One of several reasons for this is exceptions haven’t always been portable, so what was done in the past has stuck. Mozilla C++ code uses return codes to simulate exceptions. That means that while you can use try-catch blocks in JavaScript?, in C++ you should check the return code whenever you call an interface method. That return code is of type nsresult. For this reason, the logical return type, as defined in the IDL file, is mapped to an additional method parameter in the C++ code.

The nsresult type is intended to transport additional failure reasons. Instead of simply reporting failure or success, an integer type is used instead, to allow for the definition of a lot of different error codes.

There are some general result codes, like NS_OK, which is used to indicate that everything went well and program flow can continue, or NS_ERROR_FAILURE, which is used if something went wrong, but no more details need to be provided as of yet.

In addition to that, each component can request its own range of integers to define error codes that will not overlap with those failure codes used in other areas of an application. Look at mozilla/xpcom/base/nsError.h for more information.

Is it possible to edit javascript in firefox without recompile?

YES!, in most cases. What you have to do is to find your installation's omni.ja, and unzip it. Do your changes, afterwards from inside the directory you extracted omni.ja to, do:

zip -qr9XD ../omni.ja *

In the parent directoy it's a omni.ja which you can now replace with the one you found in your installation. If what you do is experimental and you don't know if it will work or not, maybe backup your installation's omni.ja first. :)

How to keep the code clean?

Many things can be done, one thing I would like hackers of the browsers to be aware of fast is the possibility to use Javascript Modules (ES6 stuff), please check out mozilla's wiki on this subject.

Plugins, Extensions, Add-ons, wtf?

Yes this can be quite confusing to begin with. In reality it's only the webextension api inherited from Google Chrome that browser third-party developers can use. However since we're so awesome we ship our own, we bypass this limit of possibilities. The reason for this is that Firefox still heavily uses the old API internally in Firefox.

We have three main categories of "Firefox API" before they introduced WebExtensions on top of it all:

  1. Classic extensions (not restartless): these will typically overlay the browser window and run code from this overlay. Since there is one overlay per window, there will be as many code instances as browser windows. However, classic extensions can also register an XPCOM component (via chrome.manifest as of Gecko 2.0). This component will be loaded on first use and stay around for the entire browsing session. You probably want your component to load when the browser starts, for this you should register it in the profile-after-change category and implement nsIObserver.
  1. Restartless extensions, also called bootstrapped extensions: these cannot register overlays which makes working with the browser UI somewhat more complicated. Instead they have a bootstrap.js script that will load when the extension is activated, this context will stay around in background until the browser is shut down or the extension is disabled. You can have XPCOM components in restartless extensions as well but you will have to register them manually (via nsIComponentRegistrar.registerFactory() and nsICategoryManager.addCategoryEntry()). You will also have to take care of unregistering the component if the extension is shut down. This is unnecessary if you merely need to add an observer, nsIObserverService will take any object implementing nsIObserver, not only one that has been registered as an XPCOM component. The big downside is: most MDN examples are about classic extensions and don't explain how you would do things in a restartless extension.
  1. Extensions based on the Add-on SDK: these are based on a framework that produces restartless extensions. The Add-on SDK has its own API which is very different from what you usually do in Firefox extension - but it is simple, and it mostly takes care of shutting down the extension so that you don't have to do it manually. Extensions here consist of a number of modules, with main.js loading automatically and being able to load additional modules as necessary. Once loaded, each module stays around for as long as the extension is active. They run sandboxed but you can still leave the sandbox and access XPCOM directly. However, you would probably use the internal observer-service module instead.

We mainly uses the classic one, but for launching the I2P router in the future we maybe need to use restartless internally - or just make firefox (without a extension) run it. People who wish to develop on our extensions are somewhat free to use the Add-on SDK API if they are afraid the ugly XPCOM monster will come and bite them, but currently we do not have any plans of using the API ourself as Meeh is now somewhat already known with XPCOM.

Privileged Javascript, wtf?

The fact is that most of Firefox's source is probably Javascript, without me fact checking now. Yes, Firefox contains quite a lot of C++ and some Rust etc.

However the glue to make all those different components together to a usable browser is privileged Javascript. It's about the same as regular, but totally different environment and APIs.

You can launch firefox/i2pbrowser with the —jsconsole argument to spawn the browser console at launch. If not, ctrl+shift+J and cmd+shift+J for OSX users.

let console = (Cu.import("resource://gre/modules/Console.jsm", {})).console;
console.log("Hello from Firefox code");

Often all you need would be imported/required in this statement:

const {classes: Cc, interfaces: Ci, manager: Cm, results: Cr, utils: Cu, Constructor: CC} = Components;

Here are some other nice snippets:

Components.utils.import("resource://gre/modules/Log.jsm");
function getStackDump() {
  var lines = [];
  for (var frame = Components.stack; frame; frame = frame.caller) {
    lines.push(frame.filename + " (" + frame.lineNumber + ")");
  }
  return lines.join("\n");
}

This one prints the extensions that's registered in firefox:

Components.utils.import("resource://gre/modules/AddonManager.jsm", null).AddonManager.getAllAddons(addons => {addons.filter(addon => addon.type == "extension").forEach(addon => console.log(addon.name, addon.getResourceURI().spec))});

This code produces a standard notification, on OSX at least, the native notification system. Not tested myself on Linux and Windows but I guess they, at least Windows, provide something similar which Firefox uses under the hood.

var myServices = {};
Cu.import('resource://gre/modules/XPCOMUtils.jsm');

//set it up
XPCOMUtils.defineLazyGetter(myServices, 'as', function () { return Cc['@mozilla.org/alerts-service;1'].getService(Ci.nsIAlertsService) });

//when you need to use it
myServices.as.showAlertNotification('chrome://branding/content/icon64.png', 'this was lazyloaded', 'this is a notification from myServices.as', null, null);

This code can load and execute files from file://, resource:// and chrome://.

let context = {};
Services.scriptloader.loadSubScript("chrome://my-package/content/foo-script.js",
                                    context, "UTF-8" /* The script's encoding */);

How can I query the I2P health service?

Just a note, the API was written quite quickly and is not pretty at all, it should change in near future so remember to check if the example bellow is up to date :) (i2pbutton with the chrome/content/i2pbutton.js should contain our live example)

let checkSvc = Cc["@geti2p.net/i2pbutton-i2pCheckService;1"].getService(Ci.nsISupports).wrappedJSObject;
  let req = checkSvc.createCheckConsoleRequest(true);
  req.onreadystatechange = function(event) {
    if (req.readyState === 4) {
      // Done
      let result = checkSvc.parseCheckConsoleResponse(req)
      i2pbutton_log(3, "I2P Console check done. Result: " + result)
      callback(result)
    }
  }
  req.send(null)

Common methods to solve problems

In firefox they use a lot of the concept "observer". We use it ourself to enable the I2P Health check, and in the future to launch I2P itself.

Here is a example on a object registering for "firefox global" events. "quit-application" is self-explained, but for "profile-after-change" it is the first event we as a plugin and not base firefox can hook ourself into.

If you want more information on why "profile-after-change" is the correct one to use, and how, take a look at Receiving startup notifications at the Mozilla wiki.

function myExt() {}
myExt.prototype = {
  observe: function(aSubject, aTopic, aData) {
    switch (aTopic) {
      case "quit-application":
        stopServer();
        obs.removeObserver(this, "quit-application");
        break;
      case "profile-after-change":
        startServer();
        obs.addObserver(this, "quit-application", false);
        break;
    }
  }
};

But to fully register you also need to edit the chrome.manifest or a equivalent file if it's in the firefox codebase. Bellow are the line we would need in chrome.manifest when it's about plugins;

category profile-after-change MyComponent @foobar/mycomponent;1

Remember consepts like the observer is kind of like interfaces, if you don't implement all functions the object is said to have, your code would throw a exception and not work as expected.

To read more about all the different notifications a observer can receive please check out Observer Notifications at Mozilla wiki.

Another common problem I want to document and bring up is preferences. Since we use them heavily, most likely you would need to deal with them as you're hacking on the browser. Here is a quick and clean read example:

Cu.import("resource://gre/modules/Services.jsm", this);
let accepted = Services.prefs.getBoolPref("toolkit.asyncshutdown.testing", false);

Recommended development preferences

Note: Not all preferences are defined by default, and are therefore not listed by default. You will have to create new (boolean) entries for them.

devtools.chrome.enabled = true
  - This is quite awesome, enable this and you get a REPL for the high privileged Javascript code that makes firefox. 

browser.dom.window.dump.enabled = true
  - Enables the use of the dump() statement to print to the standard console. See window.dump for more info.
     You can use nsIConsoleService instead of dump() from a privileged script.

javascript.options.showInConsole = true

dom.report_all_js_exceptions = true

devtools.errorconsole.enabled = true
  - Logs errors in chrome files to the Error Console.

nglayout.debug.disable_xul_fastload = true

nglayout.debug.disable_xul_cache = true
  - Disables the XUL cache so that changes to windows and dialogs do not require a restart. This assumes you're using directories rather than JARs.
     Changes to XUL overlays will still require reloading of the document overlaid.

extensions.logging.enabled = true
  - This will send more detailed information about installation and update problems to the Error Console. (Note that the extension manager 
     automatically restarts the application at startup sometimes, which may mean you won't have time to see the messages logged before 
     the automatic restart happens. To see them, prevent the automatic restart by setting the environment NO_EM_RESTART to 1 before 
     starting the application.)

dom.report_all_js_exceptions = true

Howto get started with i2pbutton

Much of our features lives here. And a great thing to notice with the i2pbutton is that you do not need to recompile firefox and so on to test your development build.

Clone the i2pbutton source, do your changes and then from the root of the repository do something like:

./makexpi.sh
cp pkg/i2pbutton-0.3.2.xpi  ~/I2PBrowser-Data/Browser/914o5i1s.default/extensions/i2pbutton@geti2p.net.xpi

But of course remember to correct paths and the version number might also have changed. Use your brain on this one, not just copy&paste :)

Howto restart firefox fast

Cc["@mozilla.org/toolkit/app-startup;1"].getService(Ci.nsIAppStartup).quit(0x12)