Building WebKit with VS2012

Sep 042012
 

I didn’t set out to build WebKit with VS2012, but that’s exactly what I ended up doing anyway…

I just built a new workstation from scratch. Clean as a baby’s memory, I thought it’d be fun to taste my own dog food. Starting with a newly installed Windows 7 64-bit and my previous walk-through I set to build WebKit.

Software Setup

  1. First, I downloaded the Cygwin downloader from here and ran it. Once complete, I ran setup.exe, gave it a target path and selected to install all packages.
  2. Next, I installed Visual Studio 2010. We really care about C++ tool chain. But other features may be safely installed.
  3. Windows SDK 7.1 was installed next. I chose to install the x64 version, of course. Again, we need only C++ for x64, but installing other stuff is fine.
  4. Now I could install Visual Studio 2010 SP1.
  5. And finally the fix to restore Visual C++ 2010 SP1 Compiler Update for the Windows SDK 7.1 (because the SDK installs pre-SP1 files).

Notes:

      Cygwin has warned that the latest version (1.7) is a major update over the previous one. This sounded more like a hint that things might break. As we’ll see, this was not a vain warning.
      The order for installing Visual Studio and updates is important.

Preparation

I decided to attempt first to build what I knew to be a good source (with a large project like this, things break fast,) so i started with r106194. After extracting the archive, I copied the environment setup script (aptly called vs2010-build-env.cmd) from the previous post. Reviewed the Cygwin source path to be valid (I had to update it because I’ve since replaced Tools folder with Bin). Running the vs2010-build-env.cmd script As Administrator, I got a bash shell.

First attempt failed with a new error:

Cygwin’s Perl package is missing the HTTP/Date.pm module. CPAN to the rescue:

Let’s try again:

Apparently sometime since I built last in April and today Brent Fulgham’s mac.com page went dead. He is the maintainer and host of WinCairoRequirements. I had to either resurrect an old version from my backups or point the script to WinCairoRequirement’s new home. I chose to start with the latter. The updated script shows the new location of the archive to be http://dl.dropbox.com/u/39598926/WinCairoRequirements.zip. But replacing the old URL with the new one wouldn’t work. Apparently DropBox doesn’t report the last-modified date of a file, which is necessary to detect updates, so a workaround is also added in the relevant scripts. Seems that I need to move to newer source revision after all.

Before giving up on the old revision (which I know builds without errors) I decided to do manually what the update script does automatically. I copied the necessary WinCairoRequirement files from an old backup and commented out the perl update-webkit-wincairo-libs script.

Now, I could see the build script complaining that it couldn’t find Visual Studio tools directory. It’s time to update the pdevenv script with support for VS2010. After replacing the pdevenv script with the one provided in my patch zip from the last post I built again. (Well, actually, I cheated: I have a newer and improved version of pdevenv script… more on that later.)

Elevated Mode

It’d be really nice if it were possible to request elevation automatically. There doesn’t seem to be a standard way of doing this, but Johannes Passing has created a tiny, but supremely useful, tool called Elevate to do just that.

Simply place the binary somewhere on your path (I have a special directory for similar utilities) and call pass the script file path as a parameter to it. There are many ways to use it. One is to create a single-line bat/cmd file with “Elevate cmd” in it. From the new prompt, one would call vs2010-build-env.cmd. Another solution is to place Elevate before the mklink calls in vs2010-build-env.cmd. The first has the advantage that you won’t be prompted more than once, but then again, that might elevate your risks as well.

Note: In the previous posts I suggested forcing Visual Studio to run as Administrator by changing the executable’s properties. However, this isn’t necessary, so long that we invoke it from an elevated shell. Because of (or thanks to, rather) the need for elevation to create symbolic link for Cygwin, there is no need to mess with Visual Studio binaries.

Back to WibKit.

Building

Running build-webkit --wincairo again throws a new error:

A quick search shows that we have duplicate entries in the environment variable. Sounds like another new Cygwin issue. Let’s see what our environment variables look like:

Seems that we do have duplicates after all. Previously Cygwin used to upper-case all environment variables, so duplicates that deferred in case only got eliminated. Cl seems to be doing something very similar, only it does detect the duplication and doesn’t silently overwrite one another. What is odd is that the lower-case versions seem to be inherited from Windows, where they are all-caps. Calling set in cmd shows all variable names to be in caps. I’m tempted to think that Cygwin is changing the Windows-inherited variables that conflict with its list to lower-case.

The following Perl snippet does the trick:

…well, not quite. Turns out MSBuild.exe, which is a background process that visual studio employs for building, needs to be killed so it would run with our modified environment variables. Otherwise, whatever changes we make to the environment variables, MSBuild.exe will ignore and all CL.exe instances will inherit whatever MSBuild.exe has. This is certainly a bug in the design of MSBuild in that it doesn’t detect and update its environment variables. Instead, it just assumes that whatever it has on the first run will be valid on future runs. Clearly an unwarranted assumption.

Let’s try building again after killing MSBuild.exe and with the new script.

Success, At Last

Here is the complete output, with Quiet log-level, for reference:

Astute readers might notice that my previous build took 49m:07s while this one is only 07m:41s. This is thanks to my new 6-core i7 3930K @ 4.6 Ghz, 32 GB 1600 @ CL9 on 256 GB Samsung 830 SSD. The previous build was on Core 2 Quad 9300 @ 2.7 Ghz, 4 GB 800 @ CL5 on 150 GB WD VelociRaptor @ 10,000 (with RamDisk). That’s an almost 6.5x difference.

The dos2unix entries are auto-conversion of Bash files that often don’t have correct line-ending, which used to break the build on some Cygwin Bash instances. This takes a couple of extra seconds but ensures that no line-ending discrepancy exists.

Latest Revision

Now it’s time to try the latest revision. It’s interesting to see what difference do 20,000 revisions and almost 1000 new files make. After extracting the archive, I simply copied the contents of my patch (2 files) overwriting pdevenv. I made sure that no instance of MSBuild.exe was running.

I might have been a bit overoptimistic, but I really was hoping it’d build. There seems to be missing source files. First assumption is that I got unlucky with the revision. After building over half a dozen revisions I was convinced it wasn’t a fluke. I checked the repository and sure enough the file is not there. Digging further I found a rather old version of the file. Circa April 2011. Further investigations showed that the file in question is a port-specific implementation. Chromium has its own implementation. But this version is the default. My best explanation is that the last major port on Windows was Safari, and they stopped supporting Windows. So that leaves Chromium and they already have a specialized implementation in their port-specific folder/project. Whatever the case, I restored the last implementation I could find, which doesn’t do anything really.

Placing the missing files gives us a successful build.

Visual Studio 2012

I thought it’d be fun to try and see how VS2012 would fare. I was encouraged after reading that the latest version of the popular IDE finally creates almost backwards compatible solution and project files. I say almost because, well, it’s not perfect. Another minor push came from the fact that Windows 8 SDK is included and DirectX SDK as well. That meant less setup hassle.

First Impressions: A Mini Review

Visual Studio certainly warrants a full review in its own right, but a quick overview will do here. First thing to notice was that it loaded very swiftly. Projects loaded faster as well. Mind you, I also thought it must be my new hardware at work. But VS2010 doesn’t load nearly as fast. Next was a small disappointment as I stared at the drab looking GUI. It was ’97 all over again. Everything is grey. Icons are grey too. Are we that desensitized by colors that we want to do away with them? I found finding commands in the menus difficult as my eyes were struggling to find anything familiar at sight without reading each entry.

First thing I did was to go to the Options dialog. To my sweet surprise, there is a dark theme! I almost exclusively use dark themes in my editors for lesser eye strain. The high contrast of black-on-white isn’t very relaxing to the eye nerves, especially after long hours. Dark themes help by lowering the contrast. Native support of dark themes is very important because some plugins assumed that the background was always light-colored. So their default choices of text color has been dark colors. This can be fixed in most cases, but some text colors aren’t configurable. I immediately switched to dark theme expecting a restart, to my surprise it instantly redrew the whole application without a single blemish. VS2010 couldn’t even change the background color of its Solution window (it wasn’t ported from Win32 to WPF).

Next I noticed that (finally) variables are highlighted when the cursor is placed on them. This has been in Notepad++ for ages. VS2012 chose to support highlighting variables within the current context. So that also means only variables are highlighted. One can’t highlight all new keywords, for example, to see all allocations at a glance. Still, it’s an improvement.

Things aren’t perfect, of course. There is always a catch. After running VS2012 on a project in my RamDisk I noticed that Intellisense stopped working. With it, highlighting went away. I tried everything and the only change is RamDisk. When the project is loaded from RamDisk, Intellisense files aren’t created or created with a minimal size. Visual Studio complains upon closing (and of course upon invoking any Intellisense comand) that it couldn’t load Intellisense files. Obviously a bug.

Another sweet feature that I noticed right off the bat is the “quick preview” (as I call it) feature. Basically, selecting any file shows the file in a single tab to the far right which is recycled. Double click to open a permanent tab or click on the “Keep Open” button on the temporary one. This works best from the Find window where one can quickly go through the found lines by the cursor and preview the lines in context without cluttering the tabs.

Btw, notice the solution loaded is VS2010 converted. It didn’t need any modifications to load (it loaded silently). I don’t know if incompatible changes are made when project settings are changed.

Building with VS2012

It was really going great for the longest time, until it didn’t. MIDL found an indefensible statement:

The MIDL version of the SDK that ships with VS2012 doesn’t like forward declaring without defining. I replaced the offending typedef with a simpler forward declaration. To automate the modification, I updated pdevenv with the appropriate Perl scripts.

Next set of errors came from the Linker. Missing functions. A quick look reminded me of VS2010 project conversion errors. In previous version of Visual Studio the project files had an entry per file and nested within that entry were the configuration-specific nodes. In VS2010 and VS2012 this has changed to a tool-specific node that has a file as its attribute. Because ignored files in the original VS2005 project had their tool set to custom, upon conversion VS2010 and VS2012 have decided that the relevant files are custom files, although they are C++ sources. To add insult to injury, VS2012 has changed the custom tool tag-name from CustomBuildStep to CustomBuild.

Next the complaints came from Tuple. This is most odd because one would expect newer versions of the compiler to have better support of the standard, not worse.

Anyway, noticing that this was indeed test project, and the errors (too many to list here) coming from gtest, I thought it fair to call it a day and let Google worry about bringing gtest up to speed with VS2012. As far as I was concerned, I build WebKit on VS2012, sans the tests, of course.

Full Script

What’s Next?

It’d be great if all the test projects also compiled. I’m sure in time this will get fixed. Another thing that I’d like to tackle is building the now-discontinued CoreGraphics on Windows. I expect this to be more of a challenge as the port rots. Still, it’s interesting. Finally, I’ll start maintaining a fork of WebKit where these fixes could be applied (yes, I tried, it’s really hard to get any unsupported code into the trunk).

Conclusion (and Patch)

Building WebKit is always challenging (and fun). Building the latest revision from scratch wasn’t trivial, but previous experience helped quite a bit. Moving to VS2012 wasn’t a breeze either, but it wasn’t nearly as hard as getting the first successful build with VS2010. Now, armed with this walk-through, one only needs to download the patch and extract it in the root of a recent revisions of a WebKit source tree for an easy and (hopefully) successful build.

Don’t shy from sharing your experience. Happy Hacking.

Building WebKit on Windows 7 with VS2010

Jan 292012
 

In a previous post I outlined a detailed description of how to build WebKit on Windows XP. My aim was to share with everyone the long and rather tricky process of building the beast that WebKit is. There are many pitfalls, gotchas and critical requirements to get from code to binary to execution. I also did away with the standard and default requirement of VS2005 and opted to go with the newer and less-buggy VS2008. Besides the improvements in this newer version, there was only 1 update to install (SP1) as opposed to the 5 or so that VS2005 required (in addition to QuickTime SDK, Windows 2003 SDK and DirectX SDK), it had a better compiler and, in my experience, caused much less problems (consider for example that newer DirectX SDK doesn’t support VS2005) to build WebKit than VS2005 did.

But that was VS2008 and Windows XP. Now, we’ll attempt to build WebKit with VS2010 and on Windows 7. This adds two groups of issues to the mix: One on the compiler/environment level and the other on the OS level. Regarding the issues that Windows 7 brings to the table, see the previous post and here. As before, I’ll build only WinCairo, but you should be well-advised to try the default – CoreGraphics.

Note: While this walk-through should be complete, obviously there are many concepts specific to WebKit and/or the tools, these I won’t repeat here. You don’t need to read the previous post, but if you find some reference that isn’t clear, search for the keyword in question in the previous post. If that doesn’t help, don’t shy from asking in the comments; I’ll get back to you.

Summary (TL;DR)

This is a detailed walk-through of getting WebKit built on Windows 7 with VS2010. For the bottom-line and an automated patch just skip the details and jump here. There is a very easy, virtually flawless, two-step, five-click procedure to get from code to binary.

Motivation

This project took my spare time during a whole week and almost two weekends. That’s an estimated 20 hours. So why do it? VS2005, which is the officially supported version, is outdated, has a painfully large number of updates and the end-user must install all its updated/patched runtime libraries and is notorious in losing/reseting the Include and Library paths. On top of that, it’s not supported on Windows 7. Since I upgraded my workstation to Win7 I couldn’t and didn’t want to install VS2005 (nor VS2008 for that matter). I could either build WebKit in a VM on WinXP (actually my old system is now turned into a cosy VM, so I could use that,) but then WebKit is too huge for a VM. The adventure of attempting a build on VS2010 was intimidating enough, considering that a single unsupported/incompatible issue could mushroom into 100s of thousands of errors that might actually require a sizable code change, which would be far more than what I bid for. To top it, I knew that Windows 7 had its own set of issues related to UAC and Cygwin’s Fork implementation, which conflicted with Windows’ randomized image mapping and other security features.

Ultimately, I wanted a clean build with the latest tools on the latest Windows incarnation. WebKit.org might not be tempted to force its development team to shell out for VS2010 or Windows 7, not to mention spend all that time to upgrade both their machines and code, but, eventually, they will. So, here it is. A complete walkthrough documenting every step of the way, from code download to environment setup to successful build. In addition, all changes to fix errors are automated with just two scripts… read on.

Environment

WebKit has a lot of dependencies and requirements. So first we need to setup everything that is expected by the code before we even try to build.

Getting the Code

As with the tip from the previous post, I avoided the Trunk and used the prepackaged sources of r106194 (but relatively recent packages should also work). While the trunk may often be broken, the nightly build isn’t guaranteed to build. To increase your odds, pick a revision of the sources that has a Mac or Windows nightly build as well. From the 7 nightly builds I picked (r105569-r106194) 2 were broken. A considerable time is wasted figuring out whether or not the errors are our fault or the source code’s.

Cygwin

The first hurdle was to setup Cygwin. Get the download instructions from the previous post. (Read and use the Cygwin downloader from here is the gist of it.) Since WebKit expects the sources to be within the user’s home directory of Cygwin, I decided to avoid placing the 100s of thousands of files and the gigabytes that a single source working-copy is within Cygwin. Instead, I can have WebKit sources anywhere and Cygwin elsewhere and get them both together using symbolic links.

My Cygwin installation is in my Tools folder, which hosts a rather large number of portable programs and scripts, including MinGW. Tools is in the root of my system drive. To make things easy for me, I’d like to have Cygwin in the system drive as well (it’s a fast drive). So the first thing is to setup symbolic links where from the original folders to where we’d like to see our environment setup. This is done using the mklink command, but before executing it we delete any directories in that path, so make sure your target path is safe!

Visual Studio 2010

With VS2005 and VS2008 the Express editions worked fine. Since I don’t have VS2010 Express, I can’t vouch the same guarantee. Regardless, it’s probably reasonable to expect it to work. At any rate, we need to install a functional version. We also need to install SP1 and Windows SDK. Microsoft deserves some scolding for releasing Windows SDK that reverts Visual Studio updates for yet another time. So, to avoid stepping on your own feet, here is how you need to proceed with the VS2010 and WinSDK installations:

  1. Visual Studio 2010.
  2. Windows SDK 7.1.
  3. Visual Studio 2010 SP1.
  4. Visual C++ 2010 SP1 Compiler Update for the Windows SDK 7.1.

Interestingly, I didn’t change the toolchain from VC100 to Windows SDK 7.1. I also didn’t install QuickTime or DirectX SDKs. It seems that the Windows and DirectX headers that come with VS2010 were enough that I didn’t have to go through any settings as I had to with VS2005. However, QuickTime’s missing SDK, while didn’t break the build, might mean that there wouldn’t be full media support in the resulting build. Actually, I suspect QuickTime SDK is needed for the CoreGraphics build. Nevertheless, that’s a minor issue that can be remedied later.

Environment Variables

Next we need to setup the WEBKITLIBRARIESDIR and WEBKITOUTPUTDIR environment variables, add scripts folder on the path and setup VS2010. WebKit build scripts expect VSINSTALLDIR environment variable, which, it seems, VS2010 installation isn’t creating. I found only VS100COMNTOOLS on my system, so I added a make-shift VSINSTALLDIR to my build environment setup script. Once that is done, we need to ask VS to setup its own environment variables and fire-up Cygwin.

I want to automate as much as possible, including the ability to run the same boiler-plate setup script for any revision of WebKit. The script is designed to be thrown into the source code folder of WebKit and run as administrator. The only missing thing is your Cygwin installation folder. But don’t worry, if you don’t provide this folder (or is invalid,) of you don’t run as admin, the script will warn you and stop.

Here is the cmd script I use to do the magic of setting up the build environment:

vs2010-build-env.cmd:

This should be straight forward. All we do is setup the Cygwin and source-code folders and the environment variables. Take your time to read it and don’t forget to set your Cygwin installation folder to CYGWIN_SOURCE_DIR.

Building

First, download the source code and extract it in any folder you like. Next, place this script in the extracted folder. What you need to edit are: CYGWIN_SOURCE_DIR to point to your cygwin folder. The other settings should be automatically set. Note: You must run this script as Administrator!

Here is a sample of the output you should expect:

If you get the prompt as I did above, then you’re good to go. But before we try to build, we need to let the build scripts know about VS2010, which isn’t really supported. Go to Tools\Scripts within the source folder and open the pdevenv file for editing. On the 18th line you should add a case for VS100COMNTOOLS and make it the first, next VS90COMNTOOLS followed by VS80COMNTOOLS. This way VS2010 will be preferred, followed by VS2008 then by 2005 if neither is found. Here is how lines 17-31 should look:

Let’s do an initial run of the build script so it’d download the WinCairo files and install them (no change is done outside the WebKit source folder).

Here is an initial run…

Great! So far so good. We downloaded and installed WinCairoRequirements (to redownload, just delete WebKitLibraries\win\WinCairoRequirements.headers) and we got VS2010 to run and try to load (and complain of) the solution file. Let’s upgrade it.

Open the solution file located at Source\WebKit\win\WebKit.vcproj\WebKit.sln in VS2010. (The easiest way to run VS2010 is to simply run pdevenv from the shell.) You’ll be prompted to convert the solution and all 40 projects. Now we should save the solution and close VS. (Alternatively you can use the /upgrade command.) Let’s go back and try to build again. Run build-webkit --wincairo from the shell. You should see the projects getting built and fail with more errors than you can count. Let’s Fix the errors.

Windows 7 UAC

Here is a generic error that’s omnipresent:

The above error actually means “CL.exe” failed to execute. It’s a general failure. The culprit here is UAC of Windows 7. To fix this issue, navigate to the VSINSTALLDIR folder, in Common7\IDE\ there should be devenv.exe (not devenv.com). Select the file, open properties then Compatibility tab. At the bottom under “Privilege Level” check “Run this program as an administrator”. Now VS2010 should run with full privileges and executing CL.exe (the C++ compiler) should work.

This time the Linker is complaining of not finding an OBJ file. Here we notice a couple of things. First, since CL no longer complains, it’s safe to assume our UAC fix worked and CL ran. Second, and more importantly, we see no CPP file compilation output before the linker error. Checking the output folder clearly shows no OBJ files. May be the output folder is different, so we need to see more details. Changing the log-level from Visual Studio (Tools > Options > Projects and Solutions > Build and Run. Change “MSBuild project build output verbosity” from “Quiet” to “Detailed” or higher) shows that CL is not compiling anything at all! After some investigation, it turns out the culprit is the parallelcl script (in Tools/Scripts). This middleware gets called from Tools/vcbin/cl.exe, which is what is actually called by VS2010 since it’s found first on the path. The purpose of this script is to parallelize the build by executing multiple concurrent instances of the compiler.

The older Visual Studio versions used to dump all command-line parameters passed to CL in a compiler response file. VS2010 isn’t using response files and is passing the params in the classical way. This behavior may be specific to Windows 7 as its shell supports a larger command-line param buffer than previous versions, but I haven’t checked either way. Bottom line is that parallelcl needs to accomodate this fact because as-is it’s simply failing to pass anything to CL, which is why nothing is actually getting compiled.

We can of course bypass this script altogether. Unfortunately, this will cost us much in terms of build time (which is significant to start with) because VS2010 (and 2008 as well) support only project-level parallelization, which isn’t much help for WebKit. The fix is to directly call the real CL. We may later try to do a better fix and support compiler-level parallelization, which is what this script was actually doing. To disable this custom script all we have to do is simply rename the CL.exe in Tools\vcbin\cl.exe.

mv Tools/vcbin/cl.exe Tools/vcbin/cl-.exe

Warnings (as Error)

Building again we see source files getting compiled. This time the errors come from the warnings. We resolved this issue by removing the “warnings as errors” options. Fire up VS2010 (which can be done by running pdevenv from the shell) then load the WebKit solution. Select every C++ solution (either one at a time, or all together) and open their properties. From the C/C++ > General node, change “Treat Warnings As Errors” to “No (/WX-)”. You probably want to do this for “All Configurations”. Save the solution and lets build again from the shell. To change settings, we’ll use property sheets (quick into here and here, more details here) which not only is the correct way to share common settings between large number of projects and configurations, but also WebKit heavily utilizes anyway.

Output showing “warning as error” and some superfluous warnings:

The property sheet file format was changed in VS2010. As with the C++ project file names, which changed from vcproj to vcxproj, the property sheet file names have also changed from vsprops to props. The reason for this is because now C++ projects are now built using MSBuild and not VCBuild. Old vcproj files, for example, contained file filters (which defined the folder structure shown in Visual Studio) as well as build instructions. New vcxproj files now contain only build instructions and filters are now in vcxproj.filters. Since property sheets contain compiler switches and flags, the format must also comform to the MSBuild format. The same is true for vcxproj files.

To disable “warnings as errors” open WebKitLibraries\win\tools\vsprops\common.props and change the code TreatWarningAsError from true to false. We also see a lot of compiler warning C4396 and linker warning LNK4221, so let’s disable them as well.

Lines 13-23 of common.props:

Fixing Errors

nullptr

VS2010’s C++ compiler supports the new nullptr keyword. This was previously simulated in WTF\NullPtr.cpp/h as class std::nullptr_t; which isn’t simulated for VS2010 and newer (WTF\Compiler.h disables nullptr simulation for _MSC_VER >= 1600). Unfortunately, the simulated class, unlike a native one, needs to be exported so that it may be consumed by other projects. JavaScriptCore exports it, and since it’s not simulated, it can’t be found. From Source/JavaScriptCore/JavaScriptCore.vcproj/JavaScriptCore/JavaScriptCore.def we need to remove “?nullptr@@3Vnullptr_t@std@@A”.

MIDL

Like CL, there is a MIDL wrapper in Tools/vcbin. The purpose of this wrapper is to wrap command-line parameters in double-quotes. What it does wrong is that it, like the CL wrapper, assumes that the only parameter is a response file. Unfortunately, VS2010 doesn’t feel obliged to pass a response file in every case, so the wrapper ends up double-quotting every parameter and thus ends up passing invalid parameters to the real MIDL. Two solutions: we can temporarily rename the wrapper file to by-pass it (it works for me) or we can improve the MIDL wrapper to support both cases. For now, let’s simply rename the wrapper to bypass it.

mv Tools/vcbin/midl.exe Tools/vcbin/midl-.exe

Link Errors

If you got thus far, then you compiled everything until WebKit proper. Your biggest milestones were JavaScriptCore, Interfaces and WebCore. WebCore, however, seems to be missing a dozen or so definitions. Turns out that during the project conversion VS2010 set the Item Type (under General settings) to “Custom Build Tool” instead of “C/C++ compiler” thereby getting ignored by the C++ compiler. The culprit here is the VS2005 project file, which has incorrect Item Type for these files, one the original project file is fixed, the conversion should produce valid output. These files are:

Source\WebCore\platform\network\curl\CookieJarCurl.cpp
Source\WebCore\platform\image-decoders\ImageDecoder.cpp

Just change their Item Type to “C/C++ compiler” from their properties and rebuild. If all goes well, you should get no error and everything should build successfully.

Wrapping up

Summary

Here is what we did so far:

  1. Install Cygwin.
  2. Installed VS2010 Sp1 (and Windows SDK 7.1).
  3. Set WEBKITLIBRARIESDIR and WEBKITOUTPUTDIR environment variables.
  4. Make devenv.exe run as administrator.
  5. Change Tools\Scripts\pdevenv to recognize VS100COMNTOOLS.
  6. Remove the nullptr_t export from Source\JavaScriptCore\JavaScriptCore.vcproj\JavaScriptCore\JavaScriptCore.def.
  7. Disable CL and MIDL wrappers (the files may be renamed or their folder removed from the path).
  8. Disable Warning as Error from WebKitLibraries\win\tools\vsprops\common.props (optionally disable warning C4396 and enable MultiProcessor Compilation).
  9. Change the Item Type of Source\WebCore\platform\network\curl\CookieJarCurl.cpp and Source\WebCore\platform\image-decoders\ImageDecoder.cpp to C/C++ compiler.

Of course it would be great if one didn’t have to make all the code changes every time they updated the source code. Also, VS2010 project and props files have a different extension and format than the older ones. This means that any changes done to these files won’t get reflected in our converted files. The fix to that is to delete the converted Solution file and VS2010 will convert the files again. But even the conversion is a manual process. Let’s automate these changes…

Automating

Every modification we make to the source code, is a departure from the mainline and therefore a pain in maintaining it. I wanted somethin maintainable. This meant minimal change. I didn’t want to add new scripts, if any at all. Not only that, but I also wanted to support VS2005/VS2008 builds as well. That is, I plan to send this patch to become part of the mainline codebase. Turns out, we only need to modify one script in the WebKit proper source code: pdevenv.

We must modify pdevenv anyway to add support for VS2010 toolchain, no escape to that. So, here is what we can also do. First, add the VS100COMNTOOLS check, then, we need to convert the project, edit JavaScriptCore.def to remove the nullptr_t reference. We don’t need to rename the CL and MIDL wrappers; it’s enough to remove their folder from the path. Next, we need to edit common.props to disable warning as error, disable warning C4396 (which is optional) and enable MultiProcessor Compilation (again optional). Finally, we need to find all .cpp files and change their type to ClCompiler.

All of the editing tasks can be done using perlre (Perl Regex). In fact, they are simple one-line scripts. The tricky part was to generate Windows Batch file with the perl calls from pdevenv, which is a perl script itself. The reason for this is because that’s how Visual Studio is called; it’s a separate CMD shell. As you can see, the regex code is pretty twisted. Here is the updated pdevenv script:

One-Command Build

Having installed Cygwin and Visual Studio, we need to change devenv.exe to run as administrator. Beyond that, just download the source code and the patch. Extract the source code anywhere you like, then extract the patch in the same folder. You should get a new vs2010-build-env.cmd file in the root of that folder and replace pdevenv in the Tools/Scripts folder.

Right-click vs2010-build-env.cmd and Run as Administrator. In the shell, simply execute build-webkit --wincairo and go read something, exercise or eat some healthy meal… Be back in an hour or so. Your station won’t be much usable during the build anyway.

Complete run of a patched nightly build for reference and completeness:

The Patch

The Patch: WebKit-vs2010-Patch.zip
Good luck and happy hacking.

Building WebKit on Windows

Sep 252011
 

Building a large and complex such as WebKit is anything but straightforward. WebKit is the core of multiple browsers, most notably Safari and Chrome. Even the official instructions are sometimes lacking. This is mostly because such large projects are moving targets and it’s hard to keep track of all the changes in the requirements. Another issue is that most developers don’t setup build systems from scratch regularly. Probably no more than once or twice during the lifetime of a project. Even blogs, articles, forums and mailing lists that discuss building WebKit are almost completely outdated.

At any rate, these notes are up-to-date as of September 2011, I hope you find them helpful.

Background

WebKit can be built in different flavors. The differences are important, because one must setup the build system slightly differently for each flavor. There are essentially 4 different flavors: Apple Windows, WinCairo, Qt and GTK. All four require external libraries to be present for the build to succeed. I have no experience building the Qt and GTK flavors. Here I’ll cover Apple Windows and WinCairo. I do not cover building Chromium.

Build Environment

There are some common requirements, so first and foremost make sure these are setup correctly. You may follow the instructions described here. Unless you want to build using VS2005, ignore the instructions on VS2005.

Here is a summary:

  1. Install Cygwin. Use this downloader. Preferably install to C:\cygwin (I’ll assume that’s where it is). Never to a path with spaces! Make sure you install: svn, curl, unzip, make, perl, python, gcc, gcc-g++, gcc4, gcc4-g++, bison, flex, gperf.
  2. Install QuickTime SDK in the default location.
  3. Install DirectX SDK. However, if you’ll build with VS2005, don’t install anything newer than February 2010. It’s worth noting that the August 2007 version builds WebKit just fine (as of r95358 with default flags).
  4. Install Visual Studio 2005 or 2008 with all updates and Service Packs. For VS2005 please refer to the updates listed here. Personally, I suggest sticking with 2008.
  5. Install Windows SDK for Server 2003 R2 or newer. See below for more details.
  6. Download the Support Libraries from Apple. Copy the file (don’t extract, just copy the zip) into the WebKit source folder.
  7. Create WEBKITLIBRARIESDIR and WEBKITOUTPUTDIR environment variables. The values must be fully-qualified Windows paths and not Cygwin/unix paths. If you don’t want to change you system-wide path, skip this and see the script below.

Windows SDK

If you have multiple Windows SDKs (as is typically the case when installing multiple Visual Studio versions) then you will have to setup your target Visual Studio to use the SDK of choice. There are two ways to achieve this.

The first method is to change the BIN, INCLUDE and LIB directories from the Visual Studio Options dialog. From Tools > Options menu, from Projects and Solutions > VC++ Directories page, set the Executable files‘s top entry to %SDK%\bin folder, the %SDK%\lib at the top of the Library files entries and %SDK%\include to the top of the Include files‘s entries. Where %SDK% is the installation path of the SDK. Note: This will change the Visual Studio paths for all instances. So don’t make this change if other projects depend on your current settings, unless you care more about WebKit, of course.

The second method is much less intrusive. We can change the environment variables in a shell instance and pass them on to Visual Studio. This is relatively easy and convenient, but will require scripting. In the Tools/Scripts folder of WebKit there is a script called pdevenv. This Perl script is responsible for creating a Windows CMD file which ultimately runs Visual Studio. The reason for this is because Bash can’t execute the scripts necessary for setting up Visual Studio’s environment.

Here is an example of a modified pdevenv script that sets up the SDK:

Note: If you already have SDK 6.0, 6.0A, 7.0, 7.0A or 7.1, you should be fine, but be warned: DirectX conflicts with 7.x SDKs, so make sure DirectX headers come after Windows SDK. If you have 6.0 or 6.0A you should have no problems building.

Building with VS2008

  1. Install VS2008 with all Service Packs and updates. (Express version needs a bit of a push to work. Please follow the these instructions.)
  2. Run update-webkit and build-webkit at least once. Regardless of the outcome, delete the contents of WEBKITOUTPUTDIR folder, wherever it is.
  3. Run VS2008 and open C:\cygwin\home\{username}\WebKitTrunk\Source\WebKit\win\WebKit.vcproj\WebKit.sln. You’ll be prompted to convert the project files from VS2005 format to 2008. Convert and save-all.
  4. WebKit projects have no tolerance for warnings and they are set to the highest level. Since VS2008 does emit some superfluous warnings (such as not inlining a force-inline function) the build will fail with these settings. As such, we need to change these setting.
    Select every project that doesn’t end in “generated”, right-click and select Properties. In the Properties window, from the top, change the Configuration to “All Configurations”. Navigate to Configuration Properties/C++/General in the left tree and on the right-side change the Warning Level to “Off: Turn Off All Warnings (/W0)” and Treat Warnings As Errors to “No”. Click OK to close this dialog.
    If you don’t see C++/General in the left tree it means you selected a non-C++ project. You may do this one project at a time.
  5. The project files are now ready. Save all and exit Visual Studio.

Build Notes

  1. Disable Anti-Virus scanners. (Can you spell incredible?) Cygwin needs to simulate Fork, which needs to clone process memory exactly. Many AV products add system hooks to scan processes in-memory. This disrupts memory layout and DLL relocation results in differences in the memory of forked processes resulting in memory corruption. This issue may never happen even with an AV running. However, if you see fork, out-of-memory or Virtual-Memory allocation failures, try disabling AV and try again. Some don’t remove their system hooks by temporarily disabling them, so you’ll need to uninstall them completely.
  2. Don’t build the Trunk. WebKit is large, complex and fast moving. Unless you like testing the limits of your sanity, you may want to stay away from the Trunk, which can be broken and problem-laden. Instead, get a successfully-built and packaged source archive from the nightly builds. At least have one recent successful build from a nightly archive before attempting the Trunk.
  3. Place Cygwin/bin before others in Path. In the system path, place CygWin binary folder after System folders but before anything else. Example: %SystemRoot%\system32;%SystemRoot%;%SystemRoot%\System32\Wbem;C:\cygwin\bin;. This will insure that if you have Perl, SVN and other similar utilities installed outside of Cygwin, the Cygwin version will take precendence. NOTE: This may break other products that rely on the non-cygwin variants, in that case, you’ll have to perform this change in a build-script (see below).
  4. Place WebKit sources in your Cygwin home directory. If you install Cygwin in the root, your home should be here: C:\cygwin\home\{username}. WebKit should be in a folder directly within your home. Example: C:\cygwin\home\{username}\webkit.
  5. Add the scripts folder to the path. The webkit scripts must be reachable at all times, makes sure C:\cygwin\home\{username}\webkit\Tools\Scripts is prepended to the path.
  6. Don’t update the WinCairo code unless you want to build that flavor. It seems that running update-webkit --wincairo is changing enough in the code that it’s rendering the Apple Windows flavor broken. If this happens and you can’t build Apple Windows, then simply delete the WebKitLibraries folder, and run update-webkit (without --wincairo!).
  7. Run update-webkit at least once. Even if you don’t need to update the code, you must run this script at least once. Note that if you’ve downloaded a nightly-build package this script won’t update the code to the Trunk! It’ll only update the support libraries and other scripts.
  8. You may have to build twice before it works. The build almost always fails the first time and after deleting the WEBKITOUTPUTDIR folder, so the build script must be executed more than once to get a successful build.
  9. Look for the first failed project to fix errors. Since the WebKit build copies many source files as well as output binaries from one project to another, it’s vital that all dependencies of a sub-project succeed before it is built. That is, if a project fails, typically due to missing headers, identifiers etc, before panicking look for the first failed project and try to fix it. Most probably the errors are due to a previously broken dependency. This can get tricky, as some projects depend on external scripts that may be partially completed and stop reporting errors on subsequent builds. In such a case you won’t see failed dependencies and the sub-project in question would be the first to fail. To make sure this is the case, delete the WEBKITOUTPUTDIR folder and build again. If you get errors, note the first error you got and build again. Repeat this until you get the same error. This is the error you need to fix.
  10. Disable warning as error. If you get error C2220: warning treated as error - no 'object' file generated errors, then follow the instruction for disabling warning as error in the Building with VS2008 section above.

Here is a bash script to setup the environment, update and build. You must set your home folder name. Save this file in the webkit folder, name it build.

Apple Windows

This flavor is the one used by Safari on Windows. The external libraries are copyrighted by Apple and aren’t redistributable. As such, the build system will require one to manually download an archive holding them from Apple after accepting the license agreement. This build, therefore, cannot be distributed on the internet with the external libraries. Having said that, one should at least try to get this flavor successfully built before attempting others.

To build this flavor follow these steps:

  1. Run the Cygwin shell.
  2. cd webkit
  3. ./build

WinCairo

To build the WinCairo flavor one had to manually download and install the various libraries, often having to build them manually or get a prebuilt version. Thanks to the efforts of volunteers now the update-webkit script automates all that. If you have built the Apple Windows flavor, you should be able to build WinCairo using the following:

  1. Run the Cygwin shell.
  2. cd webkit
  3. ./build –wincairo

If the build fails, try running the build script again.

However if you get “CF” identifier errors, consider updating the CF library manually. All public CF library identifiers and even filenames either begin or end with “CF”, so they are easy to spot. WebKit WinCairo uses a modified version of the original Apple sources called OpenCFLite. Download a recent version from the project’s page or get the trunk. You’ll have to bite the bullet and build the library yourself, update the headers and libs and build WebKit again.

To build OpenCFLite, open the solution in VS2005 or 2008 (depending on which you’ll use for the WebKit build). Choose the Release configuration and build the CFLite project. Ignore the mountain of warning and see if the build succeeded or not. If all goes well, in the root of the OpenCFLite source folder you should see a dist folder. Now copy the contents of dist into %WEBKITLIBRARIESDIR%\win overwriting existing files.

Delete the %WEBKITLIBRARIESDIR% folder and build Webkit WinCairo again. You should get a clean build.

Debugging

Now that you have a successful build, you may want to build the Debug version by passing --debug switch to the build-webkit script. You can execute this script directly (without running the above build script or update-webkit) or you can modify the above build script to the same effect.

With the debug build we can use the Visual Studio debugger or the ultra-powerful WinDbg. In Visual Studio simply open the solution file from C:\cygwin\home\{username}\WebKitTrunk\Source\WebKit\win\WebKit.vcproj\WebKit.sln, select one of the runner test projects as the startup and hit F5.

Happy hacking and don’t forget, you can contribute by submitting bug reports and sending patches and fixes. Good luck.

References

I couldn’t have succeeded in building WebKit without the help of numerous developers who shared their experience on the web. Here is a list of the most helpful and relevant pages. Be forewarned however, they are mostly outdated!

QR Code Business Card
Loading...