How to Build Microsoft Graph Engine on Red Hat Linux

Microsoft Graph Engine is a distributed in-memory data processing engine. It’s an open source project but the official instructions claim that for Linux, it only supports Ubuntu 16.04. But what if you’re using other distributions of Linux, like Red Hat? Well, it’s possible but it does take quite some effort to build the source. Let’s get started.

Prerequisites:

  1. You need 64 G of RAM. Otherwise, you’ll run into “virtual memory exhausted” errors during the build process.
  2. As of 12/11/2018, if you build the latest commit of the source, the output binary won’t work with Linux. I filed this issue and until it’s resolved, you should check out earlier commits.

Install required software packages:

You need git, dotnet, openssl, and c++

sudo yum upgrade -y
sudo yum install git rh-dotnet22 openssl-devel -y
sudo yum install gcc-c++ libgcc.i686 glibc-devel.i686

Build GCC

No, devtoolset won’t work due to ABI compatibility issue. You HAVE TO build GCC 5.4 or higher. This also takes the longest time (a couple of hours).

Please follow the official instruction. I chose GCC 7.4 and I only built C & C++. So the commands I used were

wget ftp://ftp.gnu.org/gnu/gcc/gcc-7.4.0/gcc-7.4.0.tar.gz
$PWD/../gcc-7.4.0/configure --prefix=$HOME/GCC-7.4.0 --enable-languages=c,c++

Set the right environment variables

These are necessary for building CMake and later the Graphic Engine.

export PATH=~/GCC-7.4.0/bin:$PATH
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:~/GCC-7.4.0/lib:~/GCC-7.4.0/lib64
export CC=~/GCC-7.4.0/bin/gcc
export CXX=~/GCC-7.4.0/bin/g++

Build CMake

Since CMake 3 is needed, you have to build it from its source. It’s very quick though. One thing to note is, when doing the “make install”, you have to be root. You need to re-export LD_LIBRARY_PATH

sudo -s
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:<absolutepathtoGCClib>:<absolutepathtoGCClib64>
make install

Build Graph Engine

Finally, we’re ready to build the source of Microsoft Graph Engine. Exciting?! As mentioned earlier, I checked out an earlier commit:

git clone https://github.com/Microsoft/GraphEngine.git
git checkout 4175d13a084453bd2a2bf43adf1cc4e76f9fe417
cd GraphEngine/
scl enable rh-dotnet22 bash
tools/build.sh

Voila, you have the Nuget package built under the build/ folder!

MySQL with Entity Framework Core in ASP.NET Core

.NET Core offers.NET developers a great way to create cross-platform applications. If you want to create a web app with some database support, most likely you will use Entity Framework (EF) Core and ASP.NET Core. In this blog post, I’d like to give you some tips if you choose to use MySQL.

1. Db Provider choice

Although there’re a few MySQL providers to choose from in the official EF Core Docs site, I would recommend the most popular Pomelo.EntityFrameworkCore.MySql. I tried other providers and it gave me issues here and there.

Specifically, you can use the following code to add a DbContext in your service configuration:

services.AddCors();

services.AddDbContext<KnowledgeContext>(options => options.UseMySql(Configuration[“MySqlConnStr”]));

services.AddMvc().SetCompatibilityVersion(CompatibilityVersion.Version_2_1);

And don’t forget to pass the DbContext to your controller’s constructor.

2. Use Fluent API instead of Data Annotation

This is more of an EF Core tip than a MySQL one. Certain restrictions such as foreign keys can only be specified with  the Fluent API.

3. Where to call Database.EnsureCreated

Again, an EF Core tip. DbContext.Database.EnsureCreated() is new method to ensure that the database and the required schema for the context exist. The database and the schema will be created if they don’t exist. However, in ASP.NET Core, every time a request comes, a controller’s constructor is called. So you probably want to put the EnsureCreated call in the Startup class. Please refer to https://stackoverflow.com/questions/36958318/where-should-i-put-database-ensurecreated.

Career tips at Microsoft

I have always been hesitated to write this post but since I’ve been fortunately enough to have great career mentors and I recently got to a new career stage, I’d like to share some of the career tips that benefited me over the years:

  1. Before you get to senior level, there’s only one tip: Get things Done, well, the direct words were slightly different, to quote from a Partner level mentor. Well, there’s more:
  2. Assume you can do your own work very well, start to think about what you can do to help others or the team. For example, if you observed most devs are doing the same manual effort over and over again, maybe you could write a tool to automate it and save everybody’s time.
  3. If you look around, you should always be able to find some inefficiency somewhere. Maybe the unit tests have low coverage, maybe the perf of the product is not so good in certain scenarios. Those are opportunities for you to step up.
  4. Junior devs would like to spend a lot of time. That’s a good thing. But time doesn’t equal to results. You should always focus on your time and effort on the most important things. I know sometimes it’s not very clear what the most important things are. But if you take a look at your manager’s commitment, or the business priorities of the whole product group, you should have a better picture.
  5. Always document things, especially with partner teams. It’s very easy to forget what you discussed in a meeting. Having everything documented would usually guarantee there’s no misunderstanding.
  6. Have a sense of ownership. I’m not talking about claiming or even fighting for the so-called ownership of important features or components. I’m talking about in general the sense of feeling you’re the owner of the product your team is working on. If there’s an issue or a customer question, never simply say “it’s not my feature or my bug”.
  7. Pay attention to communications and soft skills. Sometimes it’s common sense and learning how to better deal with people is a life lesson. I’m not going to talk about this in details but remember, this could be a blocker when you want to grow your career to senior or beyond.

I welcome any comments and feedback.

P.S., due to some personal reason, I didn’t write any blog post for a long time. I’ll try to be more active now.

ExpectedExceptionBaseAttribute and “Object reference not set to an instance of an object”

You’re writing unit tests with Visual Studio and you want to verify that you’re throwing the correct Exception. You could use ExpectedExceptionAttribute class but it only allows you to verify the type of the Exception. You might want to verify other properties so you decide to create a subclass of ExpectedExceptionBaseAttribute, which allows you to have your own Exception verification logic.

Unfortunately, if you’re using Visual Studio 2012 RTM (or with Update 1 or Update 2). You will run into a weird error at run time: “Object reference not set to an instance of an object”.

What’s the solution? Install Visual Studio Update 3.

You can get more details from here: https://connect.microsoft.com/VisualStudio/feedback/details/758875/-testtools-unittesting-class-deriv

Programmatically Paste Clipboard Text to a CMD Window (C# or C++)

To accomplish this, I have tried different approaches: posting WM_PASTE, SendKeys sending ctrl-v… None of the method seems to be working so well.

A friend from work suggested trying WM_COMMAND. Since Spy++ doesn’t work with cmd.exe, we had to use brute force to find the correct WPARAM for pasting.

The result is really simple, just one line code with C++:

 // hwnd is the window handle of a CMD window
 SendMessage(hwnd, WM_COMMAND, 0xfff1, 0);

And here’s the C# version with PInvoke:

 [DllImport("user32")]
 private static extern IntPtr SendMessage(IntPtr hWnd, uint Msg, IntPtr wParam, IntPtr lParam);
 private const uint WM_COMMAND = 0x0111;
 SendMessage(hwnd, WM_COMMAND, (IntPtr)0xfff1, IntPtr.Zero);

DOS Command Script: uninstall an application

The scenario is you have an application which was installed through MSI. And you only know the application name (in this case, the beginning part of the application name). Using msiexec command requires you to know the product GUID but the application might have different GUID for its different version. So here is the solution I came up with: querying the registry to get the list of product GUIDs (the format is always {GUID}), then match the DisplayName key with the application name.

Here is a sample script which uninstalls applications whose names start with “Windows Live ID Sign-in Assistant”:

for /f “tokens=7 delims=\” %%i in (‘reg query HKLM\Software\Microsoft\Windows\CurrentVersion\Uninstall ^| FIND “{“‘) do (
for /f “tokens=2,*” %%j in (‘reg query HKLM\Software\Microsoft\Windows\CurrentVersion\Uninstall\%%i /v DisplayName’) do (
set dn=%%k
if “!dn:~0,33!”==”Windows Live ID Sign-in Assistant” (
echo Uninstalling Windows Live ID Sign-in Assistant
msiexec /promptrestart /qb /x %%i /L+*v %TEMP%\wlidsvc.log
if %ERRORLEVEL% NEQ 0 echo Uninstallation failed. Check the log %TEMP%\wlidsvc.log
)
)
)

That’s it and it should work.

Control Panel Search Tip for Windows Vista/Windows 2008

In order to search for any Control Panel item, you should not use “Classic View”. As an example, if you want to turn off UAC (not recommended though), you search for “uac” with the normal Control Panel view and the option should come out.

The good news is the issue is fixed in Windows 7, with which you can search the Control Panel in any view.

DOS Command Script: Use Date/Time in File Names

Nowadays, we have different choices of script languages in Windows: VBScript, JScript, PowerShell script, etc. Coming from the old DOS world, I still prefer a simple batch (.cmd) script.

Say we have a simple sql query and we want to run it every 10 minutes and save the output to a file. We can simply have a scheduled job to run a .cmd file with the following content:

sqlcmd -i query.sql -o report.txt

If we want to keep a history of the result, it is best if we can have the date and time in the file name. So let us change it to

sqlcmd -i query.sql -o report-%date:~-4,4%%date:~-10,2%%date:~-7,2%-%time:~0,2%%time:~3,2%.txt

The above command will run perfect between 10am and 11:59pm. Between 12am and 9:59am, however, %time:~0,2% will have a space (such as ” 9″) since %time% formats hours without leading 0.

“time /t” command will always show hours with leading 0. So we have another idea:

for /f “tokens=1,2 delims=: ” %%i in (‘time /t’) do sqlcmd -i query.sql -o report-%date:~-4,4%%date:~-10,2%%date:~-7,2%-%%i%%j.txt

The only problem here is “time /t” is in 12hr format. So you could overwrite earlier results. It seems we still should use %time% and it will be perfect if we replace the space with 0 in the hours. So here is the final script which does the job:

@echo off
setlocal
set timehour=%time:~0,2%
sqlcmd -i query.sql -o report-%date:~-4,4%%date:~-10,2%%date:~-7,2%-%timehour: =0%%time:~3,2%.txt

Moving to Office

Re-orgs happen frequently at Microsoft and I am about to experience one.

I just got two Ship-Its for the product I have been working on for almost 2 years, Microsoft Equipt, which will be discontinued after April 30th. We just finished the last feature of the product: requesting refund via the home page. The feature is now RTW. All in all, I have participated the design and implementation of all two versions of the product, starting from just an idea, all the way to its sunset.

I am going to Join Office and work on a new product.