Introducing Insights for Tests/Questions

Today, we’re beginning to roll out Insights for Tests/Questions on HackerRank for Work to help you understand how your tests and questions are performing.

 

Thousands of companies who’re looking for technical talent use HackerRank Tests to screen candidates effectively and make their recruitment process more streamlined and cost effective. Now for the first time you can gain insight into how your Tests and Questions are performing and how you can improve them over time.


Test Insights

Go to any HackerRank test and you’ll find a link to Test Insights in the left pane. Note that there’s also a link to Question Insights for each question added to the test.

Selection_220
Test page

The first thing you’ll notice is aggregate test statistics on the top right of the page. These include % invited candidates who end up attempting the test, median attempt time, median attempt time to full score and median score. Next, you’ll see a distribution of candidate scores. This gives a very good picture about the difficulty of the test.

2
Test Insights

Next, you go one step deeper and look at question-level statistics. This helps you understand

  • How candidates are dividing their time across questions and whether the time limit for the test is sufficient or not
  • Whether the score of the some questions is out of proportion to the time candidates take to solve them and you should adjust question scores

Last, you get to see the conversion funnel of candidates i.e. % of candidates who qualify the screening round.


Question Analytics

To check out the Insights for questions, you can open the question list of a test (as mentioned earlier) or go to Library.

4
Library

The aggregate statistics, similar to the Test Insights, give a good idea about the difficulty of the question and the score distribution further gives a breakdown by candidates scores.

For coding questions, you see a very interesting breakdown by programming language. This helps you understand

  • What languages are most commonly used by candidates to answer the question?
  • Is the question relatively easier to solve in some languages than others? Some languages might give candidates an unfair advantage to solve a question much easily, which you might not want to allow going forward
5
Question Insights

Next, you see the usage of question over time. This tells you how long the question has been out there and whether it’s time to stop using it as it might already be popularly known to candidates.

 

We believe this is just the tip of the iceberg when it comes to empowering you to make your recruitment process more data-driven. Providing more insights around your HackerRank account usage, quantified returns of using HackerRank, industry-wide stats (including comparison to peers) and many more related things are among our top priorities today. We would love to hear your feedback on the Test/Question Insights and more about what insights will help your organisation.

Candidate and Tests API v1 Sunset Notice

Two of our oldest APIs – The Recruit Candidate API and Recruit Test API – are being deprecated with immediate effect and they will be completely shutdown on 30 April 2015. The older APIs are replaced with a much improved v2 of the HackerRank for Work API, documented here: http://apidocs.hackerrank.com/hr4wv2

 

Most of the changes are backward compatible and many users will simply need to use the new v2 end points. You can read about all the changes in our Migration guide, and if you have any questions about this or need any assistance migrating to v2, please drop us a note at support@hackerrank.com

Scheduled maintenance update

We are having a scheduled maintenance on 19 March 2015 from  09:30 UTC to 12:30 UTC. During this time the following pages will not be accessible:

  1. The test report timeline view
  2. The consolidated candidate activity view

Other than the above mentioned pages you will be able to invite and review candidates without interruption. Candidates attempting tests will not be affected in any way.

If you have any queries about this migration or any other related issue drop us a note at support@hackerrank.com

 

Faster and better Library

One of the building blocks of HackerRank for Work is the Question Library: a collection of all the questions accessible to a user. These questions are used in Tests and CodePair (our tool for technical phone interviews). We’ve just released a number of improvements to the Question Library interface to help make it easier for users to browse available questions, add them to Tests, and keep their personal Library more organized.

The first thing you will now notice is that the Library page is much faster to load. We’ve made some key optimizations and brought the page load time down by 75%. One of the reasons for this remarkable speedup is our use of pagination: you will now be able to view 10 questions at a time.

There are two other important feature enhancements in this release:

  1. Add to Test: Now, you have an easy option to add questions from Library directly to any of your Tests. Previously, this was only possible from a Test’s question listing page.

    Questions Library

  2. Archive Questions: Over time, a user’s personal Library can become cluttered. Some questions may not even be relevant anymore, or some may have just been used enough already. Users now have the ability to archive those questions. Archived questions will not be available to be added to future Tests or CodePair interviews. They are, however, still available through a separate link in the left-hand navigation bar, and can be restored to the active library at any time if needed.

Unified Candidate Timeline

If you are a recruiter using HackerRank for Work who wanted to pull together a comprehensive view of all Test and CodePair activity of a particular candidate, you would have realised that is not possible. Well, till today! We are happy to announce the launch of a simple but exciting feature that unifies all of your interaction with candidates. We do this by creating a unique page per candidate that consolidates all the activity related to that candidate, such as invite sent, test attempted, CodePair interview scheduled, etc.

Here is a sample unified timeline:

Unified Candidate

Here you can see Shikhar’s personal profile – as captured in your Test forms, a profile photo picked up from Gravatar (if available), and all Test / CodePair activity neatly shown along a timeline.

Entry Points

Anywhere you find a candidate email address as a link, clicking on the link will take you to that Candidate’s page. Currently there are four types of pages where you can find these links:

  1. The candidate search results (what you get from using the search box in the common header found on all pages of the app)
  2. Reports tables – invited, completed, and all other report tables where we display the email address of a candidate
  3. Within a Test Report
  4. Within a CodePair Report

 

Privacy and Information Seggregation

One word of reassurance before we wrap this up.

Your company’s interactions with a candidate is your data and will not be shared with anyone else, just like you do not get to see any other company’s interactions with said candidate. The candidate timeline you see is just the candidate’s interaction with your company.  In fact the candidate page URL contains a unique identifier that a candidates gets assigned for your company. Even if this same candidate has taken a test or CodePair interviews with another company they would be assigned another identifier. This logically separates candidate records across companies and protects your data.

Any feedback or enhancement requests? Drop us a line below!

Revamped Test Invitation Workflow

Inviting candidates to take a test is one of the most common activities for a recruiter on our site. Today we are launching a fully revamped test invite workflow to make this repetitive process more efficient for our users.

The new interface is shown in the screenshot below. The first thing you would notice is the entire interface has been cleaned up. We have also addressed some common concerns raised by our users. Finally we have added a few nifty new features as well!

Selection_178

Inviting multiple candidates at a time

As before you can paste a comma separated list of email addresses in the input box. With the new changes you will be able to review the added candidates and edit the list if you so wish. Duplicates are automatically purged from the list.

Selection_182

We also have a bulk upload facility, whereby you can select a CSV file containing name and email addresses in a specific format. When a CSV is selected, the invite form is populated with email ids in the file so you can review and process them.

Selection_181

Send Copy to Myself

We’ve always had a Preview Invite option that allowed you to see the format of an invite. Some of our users wanted more. So we added yet another way to keep tabs on what we send to candidates on your behalf. By checking the ‘Send copy to myself’ box you will be added to Cc: on the actual email sent to each candidate in your invite list. Beware! If you have 100 candidates in your invite list, you will receive 100 emails!

Advanced Invite Settings

Our invite flow had some rarely used settings for some advanced users. In the new interface we have hidden those behind the ‘advanced settings’ link. Be sure to check that out, though. There are some useful features, like candidate tags, that can come in handy in some scenarios.

Selection_179

Asynchronous Inviting

The biggest improvement, though, is not as visible as the above changes, but is guaranteed to be most useful to our heavy users. We have totally revamped the backend processing engine to ensure that your invite list is processed in the background. This means you can send invites to one thousand candidates at a time, and go on to attend to other things while our system processes the invite list. In the earlier flow, the browser would have hung waiting for the server to complete inviting everyone. It is likely the process would not even have completed successfully. We have fixed these problems for good.

As soon as you press Send Invites you will be taken to the ‘Invited’ page for that test, where status of invited candidates will start appearing as and when they are processed. Depending on the number of candidates you have invited, they might not all appear on the Invited page immediately. You may have to refresh the page a few times or come back to the page after attending to other pressing matters!

We are investing a lot of time into making your most common tasks intuitive and pain-free. Let us know what you think of this change, and any other changes you would like to see in the product!

Change to HackerRank’s email address

We are rolling out a change to the domain from which we send out transactional email. Once this change is fully rolled out, all emails from us will originate from support@hackerrankforwork.com. We have started this rollout and this activity should be completed by the end of month (28th Feb, 2015). To be doubly sure that our emails never go to your spam folder, please take a moment to add support@hackerrankforwork.com to your mail program’s whitelist.

Introducing SudoRank (Beta)

We are very excited to announce the launch (in closed beta) of SudoRank – our latest automated platform for assessing system administrators. You can read more about the feature on our corporate blog. If you would like participate in the beta program please leave a comment below, write to us at support@hackerrank.com, or request a demo through the SudoRank home page

Adding code stubs made easier

Time and again we have received feedback from candidates that it is a distraction to figure out how to parse program input. It is particularly irksome for candidates who choose languages like Javascript, where knowing how to read console input is virtually never a real world requirement.

We, as well as our customers, want candidates to focus on showcasing their algorithm / problem-solving skills rather than struggle with uncommon programming patterns. To that end we promote setting up Code Stubs as a best practice while creating coding questions. The Question creation tool has always allowed a way for the problem setter to do this. Today we have made a change that make the process even easier to use.   Selection_165 Before we proceed, just a word on terminology. ‘Stub’ is the function present in the Body. This is what a candidate is expected to fill out. ‘Scaffolding’ is the part of the auto-generated code that surrounds the Stub and includes code that reads test cases from STDIN and write the output to STDOUT. ‘Template’ is used for the whole thing – i.e. Stub + Scaffolding. The screenshot above is from Step 2 of a Create New Question or Edit Question workflow. The key improvements from the older workflow are highlighted for you. In brief they are:

  1. You can now view the entire code stub the same way it is shown to candidates when they take a test.
  2. There is a clear visual indicator if the problem setter has modified the system generated templates.
  3. There is an easy way to discard user edits and go back to system generated templates.

Let us know what you think!

Save Notes for Coding questions

We’re happy to announce yet another feature to simplify the coding assessment process for companies using HackerRank for Work. Going forward, as you create your own coding questions, you can add internal notes about the question, which will not be displayed to the candidates. These notes will help a great deal in cases when the team member assessing a solution is not the same as the one who created the question. Please see the below screenshots to see where this feature is available and use it in action.

Selection_155

Create a Coding Question (Step 1)

 

Selection_156

Detailed Candidate Report

 

We plan to roll this out to all other question types (besides coding) too.  Feel free to share your feedback in the comments below.