Tester’s Digest

Tester’s Digest #79: Testing Trends

2019-02-03T00:00:00+00:00

TESTER’S DIGEST

ISSUE #79 - February 3, 2019

Tester’s Digest is having a late start to the year, let’s resume publication with a look at trends in software testing and quality predictions for 2019.

Topic: Testing Trends

World Quality Report 2018-19 says that testing’s top goal is tied to user satisfaction, AI is gaining adoption in testing tools, and also demands new skills from QA professionals who need to test AI, IoT and Blockchain based products.

https://techbeacon.com/world-quality-report-2018-19-key-trends-shaping-qa-today

Joel Colantio predicts these test automation trends in 2019: easier maintenance of automated tests, and less need for it as they would become auto-healing, AI assistance, a wealth of automation tools to pick from beyond Selenium, including better record/playback; continuous testing; and vendor merges.

https://www.joecolantonio.com/test-automation-2019/

Another trends prediction includes agile, ML, DevOps, big data, IoT, notably a move from performance testing to performance engineering, test automation combined with manual testing, shortening of the delivery cycle, and tools integration.

https://reqtest.com/testing-blog/software-testing-trends-2019/

This article seconds the rise of AI (including “robots that do testing”), DevOps, and IoT, and also adds “QA as a Service” as a progressively more popular way for companies to meet their testing needs.

https://www.stickyminds.com/article/5-things-will-impact-future-software-testing

HackerRank’s report on developer skills for 2019 tells us that JavaScript is big, particularly its React framework which is overtaking Angular; hot languages to learn are Go, Python, TypeScript and Kotlin. The biggest complaint from devs is poorly written documentation (as I’m sure testers concur!) and the biggest bug in production is oddly reported to be “deployed untested or broken code” - isn’t that the very definition of a production bug?

https://research.hackerrank.com/developer-skills/2019

Off-Topic

I was delighted to come across this categorization of “difficult people on software projects” with detailed descriptions of “Problem Personalities” and specific tips on dealing with them. Read in moderation and try to avoid labeling everyone you know as belonging to one of Neil’s categories - it’s like reading a medical encyclopedia and self-diagnosing every disease. Most people we work with are not, in fact, problem personalities. I had a good laugh at the QA lineup of The Firehose, The Blamer, The Alarmist, and others.

https://people.neilon.software/

If you received this email directly then you’re already signed up, thanks! Else if this newsletter issue was forwarded to you and you’d like to get one weekly, then you can subscribe at http://testersdigest.mehras.net

If you come across content worth sharing, please send me a link at testersdigest@mehras.net

Tester’s Digest #78: Testing At Scale

2018-12-23T00:00:00+00:00

TESTER’S DIGEST

ISSUE #78 - December 23, 2018

Today’s topic is test automation in large systems, which these days tends to mean a distributed, microservice based architecture. Since Google has just about the largest system under test, a couple of posts are from their blogs.

Topic: Testing At Scale

Google selects which tests to run at pre-submit stage (ie: on a developer’s branch) and which to skip, to save resources. The selection is made by an ML model, trained on their gigantic test execution history. All relevant tests will still run in CI, but there is a cost to skipping a test which would have failed in pre-submit, since the CI build will break and revert will be required. The model can be tuned to trade off specificity vs sensitivity.

https://testing.googleblog.com/2018/09/efficacy-presubmit.html

How Google identifies which code change broke the build - the paper is heavy on statistics:

https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/45794.pdf

Facebook employs a static code analysis tool (Infer) paired up with ML-based system (Getafix) that automatically suggests fixes for common bugs, mostly null pointer exceptions, but that alone covers a lot of ground:

https://code.fb.com/developer-tools/getafix-how-facebook-tools-learn-to-fix-bugs-automatically/

All of the above is amazing stuff that made engineers on my team salivate. Of course, most of us won’t have a use for tooling of such complexity. Before moving on to a post that’s more applicable to human-scale engineering, a word of warning: you may not need microservices at all.

http://blog.jenkster.com/2018/07/microservices-check-size.html

If you are on a microservice based architecture already or considering a move to one, this post nicely lays out the testing challenges you will face, and a promising approach: consumer-driven contract testing at the API layer between services, using a framework like Pact.

https://hackernoon.com/how-to-test-microservices-with-consumer-driven-contracts-9bf5c2c05349

Off-Topic

This 4 year old can have a bright future in QA:

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=690b0543a813b0ecfc51b0374c0ce6c8275435f0

If you come across content worth sharing, please send me a link at testersdigest@mehras.net

Tester’s Digest #77: TestBash SF 2018

2018-11-25T00:00:00+00:00

TESTER’S DIGEST

ISSUE #77 - November 25, 2018

Tester’s Digest is back! Our publication cadence got impacted by conference attendance, vacations/holidays, and the SF Bay Area air quality issues, apologies to faithful readers. This issue covers the highlights from TestBash SF conference, mostly day 1 (Nov 7) as I missed day 2 due to the heavy smoke cover over San Francisco. The lineup of talks was:

https://www.ministryoftesting.com/events/testbash-san-francisco-2018

Topic: TestBash SF 2018

This blog post covers the main points of all Day 1 talks so I don’t have to!

https://medium.com/@samantacicilia/english-version-the-amazing-test-bash-san-francisco-day-1-4db594d5d2db

Another blog picks up some Day 2 talks as well:

http://angelariggs.github.io/blog/tesbash-sf

Just a few additional links worth passing on. The beautiful intro talk by Elizabeth Hendrickson was on the distinction between authority and leadership with the main message being “You. Have. Agency”. It’s not online afaict, but you can get a sense of her style from an older blog post:

http://testobsessed.com/2015/05/i-prefer-this-over-that/

The talk mentioned the importance of shaving the right yaks. In case you’ve heard the expression but don’t know its history, this is the original MIT email with the explanation:

https://projects.csail.mit.edu/gsb/old-archive/gsb-archive/gsb2000-02-11.html

The speaker made an interesting point that organizations can employ “inverse Conway” - changing boundary lines in a software system, such as APIs, can result in organizational change, natural rearrangement of teams. This post (unrelated to TestBash) has the background on Conway’s law:

https://www.thoughtworks.com/insights/blog/demystifying-conways-law

I enjoyed Adrian P. Dunston’s talk “Tester at the Table and the Tester in My Head” on creating lasting impact by passing your tester’s mindset to developers you work with, such that they develop a little tester voice in their brains. How?

By repeating yourself a lot, with catch phrases and stories.
By making a deal between QA and Dev that brings promises on both sides: QA will catch the big bad bugs, keeping the developer safe, if Dev will stick up for the testers and do “fit and finish” to avoid burying QA in a pile of itty bitty bugs. (Seems fair to me.)
By arguing correctly, starting with agreement (we have a common goal) and not trying to win.
By promoting yourself, so the tester’s voice is heard. The deck for this talk was super fun:

https://speakerdeck.com/apdunston/the-tester-at-the-table-and-the-tester-in-my-head

Paul Grizzaffi’s talk “Automation Declared Software” calls for treating test automation as a software development process, a notion I fully agree with and follow. This talk is not online yet, has coverage in live blogging notes (see below), the speaker’s other blog posts on automation can be found here:

https://responsibleautomation.wordpress.com/

When delivering his talk “Climbing to the top of the mobile testing pyramid”, Rick Clymer mentioned a weekly tip website for Selenium users, which is worth sharing:

http://elementalselenium.com/

The presenter on “Creating a Culture of Quality”, Angela Riggs, made her deck publicly available:

https://speakerdeck.com/angelariggs/creating-a-culture-of-quality-testbash-sf

Of the 99-second talks (impromptu short presentations by conference attendees), one is online, on the Modern Testing Principles which seem to be the new twist on agile QA:

https://www.youtube.com/watch?v=X384zP-ZVvQ

https://www.ministryoftesting.com/dojo/lessons/modern-testing-principles

For more talk coverage, see this set of live blogging notes:

http://www.softwaretestingtools.com/testing-roundup/live-blogging-at-testbash-same-approach-different-location

Off-Topic

A tweetstream on what is potentially the oldest bug still in wide distribution, a long and very funny read.

https://mobile.twitter.com/Foone/status/1058676834940776450

Here’s a short retelling:

If you come across content worth sharing, please send me a link at testersdigest@mehras.net

Tester’s Digest #76: Managing Bugs

2018-10-28T00:00:00+00:00

TESTER’S DIGEST

ISSUE #76 - October 28, 2018

Apropos the upcoming Halloween, our scary topic today is bugs and how to deal with them: from writing bug reports and reporting duplicates to dealing with defect clustering and deciding what percentage of bugs to address.

Topic: Managing Bugs

On viewing bug reports as “quality offers” whose aim is to improve quality of the product – which, of course, is the whole point of filing bugs, but this post suggests a positive way of thinking and writing:

https://thelifeofoneman.com/turn-bug-report-quality-offer

When should bugs be reported individually even if they seem to have the same root cause, and could therefore be called duplicates?

https://www.stickyminds.com/article/when-testers-should-consider-bug-duplicate

On defect clustering and how to turn it from a problem into a solution:

https://www.rainforestqa.com/blog/2018-04-25-defeat-defect-clustering/

Decent intro to writing a bug report. Sadly, examples are only for login, which I tend to view as too simple to be insightful, but gotta start somewhere.

https://medium.com/@Haje/how-to-write-the-perfect-bug-report-6430f5a45cd

Not all bugs are worth fixing - set a target stability rate the same way you do for site availability:

https://blog.bugsnag.com/application-stability-monitoring/

Off-Topic

Tragicomic story from an unnamed company that lost user content through a rollout/rollback of a feature that… ah, you just gotta read it.

https://rachelbythebay.com/w/2018/10/05/recipes/

If this wasn’t scary enough, dip into our older bug-themed issue from last Halloween:

http://testersdigest.mehras.net/2017/10/28/testers-digest-36-scary-bugs.html

If you come across content worth sharing, please send me a link at testersdigest@mehras.net

Tester’s Digest #75: Testing Usability

2018-10-21T00:00:00+00:00

TESTER’S DIGEST

ISSUE #75 - October 21, 2018

Let’s take a look at the intersection of UX and QA fields. I’ve seen my share of usability bugs devolve into opinion-based arguments, and have been searching for a more scientific foundation to the usability testing endeavor. Here are some resources.

Topic: Testing Usability

Laws of UX, a great read for testers and other engineers that can help form a shared foundation between UX and QA so we can file usability bugs from first principles rather than by gut feel:

https://lawsofux.com/

Common usability bugs in web applications:

https://www.stickyminds.com/article/testing-usability-standards-our-customers-expect

What a usability bug might look like:

https://offbeattesting.com/2017/02/14/when-your-workflow-is-a-bug/

Human experience testing with the example of wearable devices:

https://www.stickyminds.com/article/testing-wearables-human-experience

Cognitive biases in user research (which are also worth keeping in mind for QA):

https://www.mindtheproduct.com/2018/08/cognitive-biases-the-questions-you-shouldnt-be-asking-by-cindy-alvarez/

If Satan designed a form to input your phone number - pure satire but jostles the brain for usability testing ideas.

https://uxdesign.cc/if-satan-was-a-web-designer-dc5cdf06dff9#.vj6c4pjzp

Off-Topic

Not completely off topic for UX, but tangential… Interesting take on the product being a function that transforms inputs (is defined by client) into outputs (defined by Product Management), the function’s body being the domain of UX. I liked how this approach would make it nicely testable.

http://www.feltpresence.com/functions.html

If you come across content worth sharing, please send me a link at testersdigest@mehras.net

Tester’s Digest #74: Tester/Developer Collaboration

2018-10-14T00:00:00+00:00

TESTER’S DIGEST

ISSUE #74 - October 14, 2018

Last time we looked at the setup where you have no QA engineers, only developers. Now let’s consider the more common configuration where you have both, and want to integrate testing into the dev process such that happens earlier (shift-left) and in a more collaborative, efficient way.

Topic: Tester/Developer Collaboration

Partnership between test automation engineers, manual testers and others in the org brings value:

https://techbeacon.com/test-automation-engineers-how-build-partnerships-matter

Integrating testing with the development team by adding a 3rd person (tester) to a pair-programming duo to test changes early results in very good first-time quality.

https://www.techwell.com/techwell-insights/2018/03/elevate-code-quality-integrating-testing-and-development

Why a dev team may push back against inclusion of testers in their process, and how to overcome this resistance with empathy and incremental changes:

https://www.forbes.com/sites/forbestechcouncil/2017/12/08/pushback-against-dedicated-qa/

How do you determine the necessary tester headcount?

https://www.techwell.com/techwell-insights/2018/09/looking-beyond-tester-developer-ratio

What is Hypothesis Driven Development, and how it would promote more rigorous testing while also allowing to move fast.

https://launchdarkly.com/blog/hypothesis-driven-development-for-software-engineers/

How Product Manager’s role fits with QA - partly specific to using RainforestQA testing platform, most points apply more generally.

https://www.rainforestqa.com/blog/2018-04-05-how-product-teams-use-rainforest-qa/

We testers are often bringers of bad news; is there a better way to deliver the message? Some debatable points here including the “shit sandwich” style of delivery that I do not endorse, but good food for thought:

https://medium.com/@fmsReliability/delivering-the-bad-news-safely-9c2fc56c7e4d

Off-Topic

Learn Bash usage and syntax (and underlying concepts such as processes and file descriptors). The guide is a work in progress. Beginners will get to a pretty good level by going through the 4 chapters that are in place. Experts should consider contributing to the GitHub project to advance this thing!

http://www.bash.academy/

If you come across content worth sharing, please send me a link at testersdigest@mehras.net

Tester’s Digest #73: Developer Owned Testing

2018-09-30T00:00:00+00:00

TESTER’S DIGEST

ISSUE #73 - September 30, 2018

What does quality process look like when you have no QA engineers, only developers? Some brave organizations have undertaken this experiment so the rest of us may learn. My firm belief, born of experience, is that testing is a skilled job, which like any other professional occupation benefits from specialization.

Topic: Developer Owned Testing

If you have both QA testers and developers performing testing activities, how do they fit together? This post references advice from Lisa Crispin who advocates developer/tester pairing and mob testing, and brings up the question of mutual trust.

https://www.joecolantonio.com/2016/07/28/getting-testers-developers-work-together/

If you don’t have dedicated testers at all, how does that work exactly? This post goes through details of task distribution, communication, and responsibility.

https://techbeacon.com/two-years-no-testers-what-i-learned

Should test automation be left to developers? This author leans toward ‘no’, with some well reasoned arguments.

https://www.ontestautomation.com/should-test-automation-be-left-to-developers/

A similar argument is laid out here. Amusingly, although the post promises a balanced approach, the argument for developer-based testing is well wrapped in caution tape.

https://techbeacon.com/qa-necessary-or-should-developers-do-their-own-testing

Developer-owned testing in practice, with an example from company Guru which uses RainforestQA platform to enable their developers to author end-to-end testcases.

https://www.rainforestqa.com/blog/2018-06-05-should-developers-own-quality/

Regardless of testers being in the picture vs developers taking on that job, here’s how to get to a culture of ownership of builds and proper engineering response to broken builds in CI: communication, fast feedback loops, and good tooling.

https://medium.com/@squarecog/its-important-and-they-aren-t-doing-it-578090bb6876

Off-Topic

How asynchronous message architecture works, illustrated by ordering coffee at Starbucks:

https://www.enterpriseintegrationpatterns.com/ramblings/18_starbucks.html

If you come across content worth sharing, please send me a link at testersdigest@mehras.net

Tester’s Digest #72: Testing Practices At Tech Companies

2018-09-16T00:00:00+00:00

TESTER’S DIGEST

ISSUE #72 - September 16, 2018

It’s been half a year since we looked at who’s doing what with regard to QA practices in the tech industry, about time for another peek. The older Tester’s Digest issues on this topic were:

http://testersdigest.mehras.net/2018/04/22/testers-digest-55-testing-practices-at-tech-companies.html

http://testersdigest.mehras.net/2017/05/28/testers-digest-17-testing-practices-at-tech-companies.html

Topic: Testing Practices At Tech Companies

Spotify’s QA strategy is based on prioritizing long-term reliability, collaboration of QA with devs and PMs, and targeted test automation:

https://www.rainforestqa.com/blog/2018-04-20-how-to-build-qa-strategy-like-spotify/

Testing practices at a television platform YouView:

http://www.devopsonline.co.uk/head-of-test-at-youview-gives-tips-to-ship-good-quality-code/

How online gaming platforms handle scheduled maintenance (and associated QA):

https://www.pcgamer.com/what-happens-when-online-games-go-down-for-maintenance/

Fault injection testing in production at LinkedIn:

https://engineering.linkedin.com/blog/2018/05/linkedout--a-request-level-failure-injection-framework

Netflix posits that a full cycle developer thinks and acts like an SWE, SDET, and SRE; testing is included in developer’s duties. Centralized tooling team supports developers in their work toward reliability and performance.

https://medium.com/netflix-techblog/full-cycle-developers-at-netflix-a08c31f83249

Crowdsourcing bug discovery (particularly for security vulnerabilities) is a new industry trend, with some companies like Bugcrowd and HackerOne acting as connecting platforms.

https://www.technologyreview.com/s/611892/crowdsourcing-the-hunt-for-software-bugs-is-a-booming-businessand-a-risky-one/

Very specific description of QA process at Netchex, a payroll/HR service provider. They use Microsoft VSTS instead of JIRA, and RainforestQA platform for testcase management and execution.

https://www.rainforestqa.com/blog/2018-06-12-how-netchex-uses-rainforest-microsoft-vsts/

Off-Topic

Eulogy to Jerry Weinberg, quoting his classic books on software management and quality.

https://www.techwell.com/techwell-insights/2018/08/importance-people-software-tribute-jerry-weinberg

If you come across content worth sharing, please send me a link at testersdigest@mehras.net

Tester’s Digest #71: Debugging

2018-09-09T00:00:00+00:00

TESTER’S DIGEST

ISSUE #71 - September 9, 2018

Debugging technique is something people tend to learn on the job, as you’re not likely to find courses on “Debugging Theory” or practical seminars, which is too bad. Our current issue covers some approaches and a couple of specific debugging tales, with more available in the earlier Tester’s Digest:

http://testersdigest.mehras.net/2018/01/13/testers-digest-43-debugging-tales.html

Topic: Debugging

Testing vs debugging, and some debugging tips:

http://www.abodeqa.com/differences-testing-debugging/

Why having a screenshot of the user’s problem is highly valuable for the developer’s debugging process:

https://lotsog.blogspot.com/2018/03/a-screenshot-speaks-thousand-words.html

A great discussion of differential diagnosis, a technique used in the medical field and directly applicable to debugging / troubleshooting, particularly when it’s not a solo endeavor but involves a team (when doesn’t it…)

https://blog.danslimmon.com/2015/10/19/troubleshooting-chatops-ddx/

From the same author, a strong recommendation to test with the aim of falsifying a hypothesis, not with the aim of confirming one:

https://blog.danslimmon.com/2016/09/07/falsifiability-why-you-rule-things-out-not-in/

Getting the bugs to reproduce can be the hardest thing. Fun story here of a Windows related bug, hunted down by the Israeli Army, and resulting thoughts on how to get the debugging data you need from real production bug occurrences.

https://blog.sentry.io/2018/06/13/comedy-of-errors-rookout

Neat overview (by the above Rookout guys) of different approaches to debugging production issues, from naive “Hail Mary” to intentional breaking things and reverting them, from monitoring to observability. The aside on “TDD cult” made me laugh.

https://www.rookout.com/the-5-approaches-to-production-debugging/

A complicated story of debugging a slow ssh connection to a machine:

https://rachelbythebay.com/w/2018/03/16/slowroad/

How to debug memory leaks in Node.js and in Java:

https://www.toptal.com/nodejs/debugging-memory-leaks-node-js-applications

https://www.toptal.com/java/hunting-memory-leaks-in-java

Off-Topic

GitHub’s announcement of “Project Paper Cuts” wherein the company shall start fixing annoying usability issues tickled me funny. Apparently, bug fixing qualifies as “exciting new ways”.

https://blog.github.com/2018-08-28-announcing-paper-cuts/

If you come across content worth sharing, please send me a link at testersdigest@mehras.net

Tester’s Digest #70: AI For Testing

2018-08-26T00:00:00+00:00

TESTER’S DIGEST

ISSUE #70 - August 26, 2018

Application of machine learning / artificial intelligence techniques to the field of software testing is a hot area these days. Let’s try to look past the hype to specific practical uses of ML/AI in testing, and consider the promise and the limits of this approach.

Topic: AI For Testing

Deep theoretical analysis of how ML could be used to develop testing tools that learn. The first post covers supervised/unsupervised vs reinforcement learning and its reward model. The second one goes deeper into Markov decision process as applied to a “gridworld” of features and bugs. The author illustrates (with Pac-Man example and working Python code) limitations and reasonable expectations of AI-based testing. Worth a read!

http://testerstories.com/2018/06/framing-automation-based-ai/

http://testerstories.com/2018/08/can-an-ai-become-a-tester/

Examples of practical applications of AI in visual testing by Applitools, using ML for test failure analysis at Dell, “spidering” tools like Mabl that auto-generate testcases after learning your app, and then run the tests by diffing their results from expected output which was learned (like Testim).

https://www.joecolantonio.com/how-ai-is-changing-test-automation/

https://www.joecolantonio.com/7-innovative-ai-test-automation-tools-future-third-wave/

Diffblue is an AI-based testcase generator that uses reinforcement learning and solver search techniques to make the ML system go beyond the training set of test examples, generalizing so it can produce new unit tests, given a codebase (only Java for now). Another AI-based tool, Security Risk Detection from Microsoft, uses constraint solving ML to increase test coverage through “whitebox fuzzing” of inputs.

https://thenewstack.io/ai-machine-learning-can-help-test-code-faster/

Newcomers like Test.AI promise AI-powered testing through the GUI that automatically identifies testcases for web page elements and user workflows, and can tolerate frontend changes. That’s a beautiful vision, let’s see if the well-funded startup can get there. Their older post (under Appdiff name) explains how they train ML to recognize app state labels from 300k labeled screenshots.

https://techcrunch.com/2018/07/31/test-ai-nabs-11m-series-a-led-by-google-to-put-bots-to-work-testing-apps/

https://medium.com/ai-for-software-testing/ai-for-testing-identifying-app-state-92a8c89a0216

Gaming company uses AI to proactively detect bugs as developers write code:

http://www.wired.co.uk/article/ubisoft-commit-assist-ai

Netflix uses data science technique of predictive modeling to locate content that’s more likely to be buggy (poor recording, audio/video mismatch, subtitle snafus) so it can be tested by humans:

https://medium.com/netflix-techblog/optimizing-content-quality-control-at-netflix-with-predictive-modeling-712281658ab9

Off-Topic

My beloved crowdsourced testing service, RainforestQA, hasn’t written a recent public post about the details of ML that is powering their framework; they have, however, written about the deficiencies in QA process on the Death Star.

https://www.rainforestqa.com/blog/2018-06-19-bad-qa-killed-2-million-imperials-death-star-post-mortem/

If you come across content worth sharing, please send me a link at testersdigest@mehras.net