"Some birds aren't meant to be caged, their feathers are just too bright"- Morgan Freeman, Shawshank Redemption. This blog is from one such bird who couldn't be caged by organizations who mandate scripted software testing. Pradeep Soundararajan welcomes you to this blog and wishes you a good time here and even otherwise.

Wednesday, March 05, 2014

All testing done should be...

All testing done should be...
  • Risk based : Caring to not  lose users, money, name, market share, brand value, customers, people
  • Business driven : Focused on goals that business wants to achieve
  • With care for users : really caring for them and not masturbating about caring for them
  • To improve sales :  focusing on conversion, funnel, lesser cost per customer acquisition
  • To increase profitability : to be more efficient while making the same money
  • To make customers happy : by just allowing them to do what they intend to with the software

Does your testing fit the above pointers?

I think tests, heuristics or your lovely test cases should be classified according to the above categories other than your favorite little bunnies of functional, sanity, regression.

If, all your testing does, is to test for functional stability and not care for the above pointers - your testing is of extremely low value and someday nobody would care for you and your test cases. Not that you don't know.

It was intentional to use the word "should be" because the pointers I gave are meta layers to why testers exist or need to exist. 

Wednesday, February 12, 2014

Fake Agile - Test Consultant

I call myself a Fake Agile - Test Consultant. All consulting I have done in the space of Agile never exposed to me what some people over the internet seem to be talking about.

When I first read the Agile Manifesto I was excited. It sounded a lot like Context Driven Testing to me. I was hoping that tons of problems I faced in my career with poor software engineering practices would go away and people would be more happy developing and testing software. I saw more stressed faces or more irresponsible faces.

In reality, all places I have been exposed to in India is running Fake Agile. Consulting such teams, it would not be true if I call myself experienced in Agile. After working with those teams, I realize some Waterfall model teams I worked with had better maturity in being Agile and agile although they didn't call it that way. Even Natural Agile teams suffer with people wanting them to be Fake Agile.

I believe in collaboration. Even before I read Agile Manifesto. I believed in People over Process and Fake Agile makes me even more believe People over Process.

As a part of my consulting Fake Agile Teams, I get them to a point where they recognize and say "Yeah, this ain't agile". I end up creating a good foundation for the Real Agile Consultant to come and do the job they are good at. 

Thursday, February 06, 2014

Pregnancy Analogies - Part 2

We look at test case pass and fail to make shipping decisions

There are pregnancy tests whose results turn out to be false positives and false negatives. In the pregnancy context, it would be very disappointing if the result is false positive or false negative, especially if the expected was an opposite. In India, I have seen and heard women who cry out of pain when the result turns negative when they want it to be positive. Similarly, teenagers (and those who think they have had enough kids) expecting a negative and false positives could drive them crazy causing a psychological stress.

A test case pass or fail can be a false positive or false negative.

Skilled software testers are aware of the existence of false positives and false negatives hence they investigate every bug they find or if software "works" despite the test case pass.

When I do strategy for testing, my strategy involves - doing tests to determine false positives and false negatives. Building heuristics (test ideas) and oracles (or expected results) that cross check results without causing an over head to my test progress is an approach that I use for my testing.

For instance, in Moolya when we were designing a strategy for testing a mobile app for a banking application - we decided to put tests in a specific sequence (that I cannot reveal) to test for false positives. We were able to test for data accuracy while it could look like simple functional tests.

Friday, January 17, 2014

A test passes or fails through what?

  1. 1.
    move or cause to move in a specified direction.
    "he passed through towns and villages"
    synonyms:goproceedmoveprogress, make one's way, traveldriveflyMore
  2. 2.
    go past or across; leave behind or on one side in proceeding.
    "on the way to the station she passed a cinema"
  1. 1.
    an act or instance of moving past or through something.
    "repeated passes with the swipe card"
  2. 2.
    a success in an examination, test, or course.
    "an A-level pass in Music"

If your meaning of Pass is Noun(2) then your notion of Pass in Software Testing maybe weak. If you were to think about Pass in Software Testing as Verb(2) then you are giving yourselves an awesome power to your testing.

Frequently asked questions

Q: How many tests passed?
Q: How many tests failed?
Q: Did that test pass?
Q: Did that test fail?

Not so frequently asked questions

Q: What does a test need to pass through to be called as pass or fail?
A: A test needs an expected result or an oracle to pass through to determine pass or fail.

Q: How does Pradeep think when he exploratory tests?
A: He constructs a test (idea - a heuristic) and runs them across several oracles allowing himself to find more problems per test (heuristic) run.

Q: How does this help Pradeep achieve better test coverage?
A: If he were to run the test with just one oracle (or expected result, you dummy) he would achieve limited coverage and limit himself from finding the bugs he can per test run.

Q: Can you give an example?
A: Sure.

Let us take a test that needs to be run to determine if the GUI writes to the database when a Submit button is pressed. The most obvious oracles is - the values provided in the GUI by the user should be written to database. This can be considered a Requirements or Functional Oracle. The time it takes to write can be another oracle. My test may pass through Requirements and Functional but may not pass through Time Oracle. If it fails the Time oracle then I need to investigate if this is about my internet connection speeds or about the code or about the server. That investigation can help me to make recommendations of what speeds of internet are required to make the write faster. Now, if there is a time out implementation to the write command then I could try delaying it and seeing what happens to the data on the GUI considering that the user has spent 20 minutes typing into several forms before hitting the submit button (which we always do. Always. Let me repeat - ALWAYS). If that fails, then we have tested the test against Recovery / Error / Data Retention / User oracles.

All this takes hardly any noticeable time. I pass through a test through so many oracles to make my Pass a Power Pass. Scripted tests are Power-less tests because there are one (or maybe two) oracles at max people have to pass it through to determine Pass/Fail.

My tests passes or fails but not the way you think it passes but the way I see it pass through my powerful construct of oracles.

Monday, January 13, 2014

Pregnancy Analogies - Part 1

Everybody is pregnant with ideas all the time. Women are gifted with a capability to be pregnant beyond ideas.

In 2010, I had the divine opportunity to learn about pregnancy through my wife who was carrying our first child. I couldn't stop my mind from co-relating pregnancy to testing software. I have been trying to crystallize my ideas for a long time (our child is more than two and half years old now) and this blog post was lying in draft version since early 2013. A long journey to get here.

When I attended Anne and Rob Saborin's talk in CAST 2008 - Testing Lessons from Labor Room, I had no way to co-relate what they were talking because I did not know about pregnancy or labor pain as much as I know after I experienced it through my wife. I wish, I could listen to that talk again.

Here is my version of ideas about pregnancy analogies for software testing. I want to structure it as questions (or situations) and answers (or responses) to it. Software testing is very simple to understand and this is why the world doesn't get it fucking right.

We are not ready for testing!

Sure. Not ready for what kind of testing. Not ready for testing the heart rate of the baby or its movements? Oh, you mean, you are in initial stages of pregnancy?

What about tests for healthiness of pregnancy? You maybe pregnant however sometimes there could be pregnancies that could lead to health risks. Ectopic Pregnancy is one such example. If you do not test for this (and that should be done much before your baby starts to take shape) , you may end up risking the mother and the child. You want to know about it earlier than later.

All tests are green. Can we push this code to production?

A doctor does not just look at a specific scan result to make a decision for C-section.She analyzes the patient history, current condition, determines what is safe for the child and the mother to make a decision for C-section.

It is a human who assesses all angles to make a decision and not just test results. Test results support the human to have more information when making decisions. Tests that bring in many kinds of information is important to good decision making. Having just one type of test keeps a different risk unaware.

When that successful company doesn't hire testers, why should I?

Delivering a baby is a life and death experience. There are many people who deliver babies at home or in situations where they are far away from hospital. The human race has not vanished because there were no hospitals or advanced technologies during the early days of civilization. Hospital and having doctors around has increased the survival rates, safety and health. Some babies have jaundice after birth and some babies develop fever post birth. Some babies like me don't cry the moment they are born. Having specialist doctors by side makes you focus on your baby than worry about what next.

Want to contribute your pregnancy analogy for testing/development/software? Write to me to my email id pradeep.srajan at the famous mountain view based gmail provider dot com and I shall mention your name and give the due credit if I like your idea in the upcoming parts. 

Saturday, January 04, 2014

2013 - the year of change

Happy New Year! For a very long time, I couldn't blog here (or anywhere). This has been the longest gap I have had in my blogging life. I thought I would not get back to this blog but thanks to Jokin Aspiazu, Mauri Edo, Issi Hassan and my fellow Hindustanis, Rahul Gupta and Sharath Byregowda to re-inspire me to blog. 2013 was THE YEAR OF MY LIFE so far. It made me change a lot of things. Personally and professionally. I have had a great start to 2014. If I am a changed man, it shall reflect on the way I think, blog and execute ideas. I have done a lot of good things that is keeping me busy in ways that is exciting to me. I am all set to really do what I have been saying I will do. Not the same way I did things in the past. 

I have disabled comments on this blog. I don't have time for it anymore. I just want to write my thoughts and leave it. I have benefited a lot from responding to comments but I am selectively choosing how I spend my time. I would love you to talk to me though. I am going to be at some conferences and you are free to drop me a line.

I am hoping to make my posts to be shorter than how it were in the past. Unless the context demands me to go deeper into something, I will try to keep it short. Thank you all for coming back and reading this. I shall continue to appreciate the time you spend here and I shall as always work hard and sincere to make it a good experience.

I would like you to follow me on Twitter, Quora and Linkedin, if you haven't yet done that. My passion to write, remains. My passion to help, remains. That won't change.

Friday, May 10, 2013

Emailing skills and its power to influence for software testers (no, just me)

No matter how much you probe people who are successful they won't tell you some secrets - not because they don't want to tell you but because they don't all the time recognize the smallest most ignored thing they did great were the secrets.

To me, when people talk about testing skills, they often leave out writing skill. They don't recognize it. Even if they do - there are several forms and forums where testers write and their writing has to vary accordingly to help them influence the project and decisions. Most of us know about bug reporting writing. I don't know of many people who give importance to emailing skill but I attribute emailing skills to be a major contributor to my success so far. In Moolya, we have emailing skill training to everybody and we constantly review good bad and ugly emails to help our folks be better than some of us. At all levels, we keep reporting bad practices on email. Parimala has even used some of my emails to show bad examples in signature section in emails and why it matters to have a signature that helps people reach you when needed.

I was looking into some emails and found a consulting assignment email that I had sent to a client I worked for in 2009 and thought it would be a good idea to put it up for the world. When Parimala reviewed the emails she thought it was brilliant and finally has something to show to testers in Moolya from on the good example side. I asked if I should publish this on my blog and she instantly liked the idea. So, here it goes:

The way I email daily updates is, I go to the previous sent email and send it as a reply to all adding the current day's update. So, if you were to receive a day 12 report, scrolling down, you will see all other day updates. I don't want to give extra gyaan but whatever you can take away from the below email, take it. 

Product information, people name, customer names and other confidential information have been changed to not breach the NDA I signed with them. 

On Thu, Oct 7, 2010 at 11:18 PM, Pradeep Soundararajan <PSoundararajan@Bravo.com> wrote:

Day 14:

         I couldn't sleep yesterday night after finding those issues. I was excited and continued testing till about 2 AM from the guest house.
         Its pretty exciting to be finding these issues and hoping its of value to you.
         I discovered yesterday that Firefox, the application we allow itself is a culprit that can help users bypass the security. Just launch firefox, type c:\ in the address bar, enter, you have access. No policy blocks.
         The other important issue that I have reported includes user being able to change the settings of McAfee virus scan itself. I was able to exclude my desktop to be scanned, downloaded a test virus and it resided without being cleaned or deleted. I could initiate a scan of registry entries, folders.
         By this, I have kind of violated almost all policies set.


         Will be end of the contract day
         I am hoping to report a few more issues that I have been investigating today.
         I shall publish to you my test strategy, test coverage, analysis, suggestions, experience report. As this would take some time to do it, I may send this early next week.

-- Pradeep Soundararajan

On Wed, Oct 6, 2010 at 11:46 PM, Pradeep Soundararajan <PSoundararajan@Bravo.com> wrote:
Day 13
         Continuing the investigations on Deskpro, I figured out more vulnerability today. These are even more serious ones than yesterday's reports.
         I have access to all drives now C:\ Q:\ F:\ this time not over command prompt but through 7Zip
         I could write into these drives. I have left a file C:\TRU\ called pradeephackedthis.txt
         I could copy all visible data to me to my local folders
         I could view individual user accounts, settings, what they have on their desktop, what they have browsed, what cookies are set on their system. 
         I can go write a cookie into their system and steal all data without their knowledge.
         I could install applications into their login. For instance, I put a zip file into Startup folder of users that when opened contains about 50 MB of data in it. There is a way to deny users with space.
         Even if a user discovers it by accident, rest is going to be excitement for him and disappointment for us.
         I could initiate an install of applications with no restrictions from the GUI itself. In fact I did end up reinstalling Tomcat.
         I feel like a super admin without needing the password or login for it.
         The thing to note is: The policy did block me but not all times.
         There are other basic functional issues as well that will be reported.
         These issues will be reported in Jira tomorrow with necessary screenshots and further investigation

-- Pradeep Soundararajan

On Tue, Oct 5, 2010 at 10:43 PM, Pradeep Soundararajan <PSoundararajan@Bravo.com> wrote:
Day 12:
         I continued testing on Deskpro focusing on functional & security issues
         Users email ids, login ids can be accessed: this is privacy policy violations. Customers may sue for privacy violation.
         Lots of open source software for windows desktop are bundled in zip file, when downloaded, all exe run inside zip despite policy restrictions
         Command prompt clones run and allow access to C:\ drive
         I could write files in C:\Windows & C:\Program Files & System32 folders
         I could see the admin policy settings
         Apart from these, I have installed software in C:\Program Files which reflects in not just my account but all users connecting to SRIVER11.
         So, if 50 users are installing 10 applications, every user may see 500 applications installed in his login
         Continue testing Deskpro / Mutro
         Investigate and report issues found today in Jira

-- Pradeep Soundararajan

On Mon, Oct 4, 2010 at 11:05 PM, Pradeep Soundararajan <PSoundararajan@Bravo.com> wrote:
Hi All,

Greetings! Hope you had a great weekend and having a good week ahead.

Day 11:
         The netbook I had where I was able to delete the Bravo folder had some issues even after reflashing it. So, about half of the day was spent on waiting that I'd get the netbook in the next 15 minutes. My mistake, should have gone immediately to Deskpro from nLive.
         Post afternoon, I started testing Deskpro from nLive and found some critical issues related to privacy of registered users. I could get access to all thousands of registered users of entire network and their email ids.
         Beyond that, I could install applications and it gets reflected in other people's account as I was able to install it in the C:\Program Files of SRIVER11. So, when logged in with admin privileges from R account, he could get the applications I have installed. This is again, in my opinion, something critical.
         Apart from these I have reported a few functional and Usability related issues.
         I am hoping to get the netbook again tomorrow, even otherwise I have Deskpro through nLive.
         As a side note, I had some amazing discussion with Architect & insights about Mutro, Deskpro & iDrive. 

-- Pradeep Soundararajan

On Fri, Oct 1, 2010 at 5:17 PM, Pradeep Soundararajan <PSoundararajan@Bravo.com> wrote:
Hi All,

Day 9:

         QA Review meeting: Thank you for briefing me on what you want me to focus on going forward. This was important for me to be of value to you.
         Focus is on functional problems for 2.0 October release.
         Prior to this & post the meeting, I have reported the functional issues I could find on Mutro

Day 10:

         Testing for functional problems and user based scenario problems on Mutro + iDrive
         Reported a couple of issues on functionality.
         Reported a blocked where after successfully launching iDrive I could move the /home/Bravo folder to trash, making all applications in Mutro to fail.
         I have reported about 19 issues over the last two days and most of them being functionality related.
Week 3:

         Focus on Functionality of iDrive, Mutro and Deskpro
         Find as many important problems as possible and report all of them
         Touch base with Architect and team on a constant basis.


On Wed, Sep 29, 2010 at 7:05 PM, Pradeep Soundararajan <PSoundararajan@Bravo.com> wrote:
Hi All,


         Worked entirely on netbook
         Got the test login from Admin Personnel for Server 2008 and explored on it for an hour.
         Covered areas: toolbar options, zones, user scenarios
         Found issues related to Usability, Functionality & User expectation violation
         Reported most of the issues confirmed on Jira
         Rest of the issues that need investigation shall be reported tomorrow.


         Focus of testing Mutro shall continue
         Have asked Admin Personnel for a few policy changes that would bring Deskpro closer to what the users might be using it in future to continue my tests on it.
         QA Progress meeting with PM, TL, Architect & Admin Personnel
         Shall be leaving office in the afternoon

-- Pradeep Soundararajan

Resending the same with updated Subject line (Day 7 instead of Day 6 as per previous email). Sorry.

Hi All,

What I loved:
         I loved the idea of showing what's happening within each zone in the preview mode. For instance I initiated Deskpro and before it completely launched I opened a new zone and went to My Zones, I could see what was opening / happening inside Deskpro in the thumbnail view. 
         That is a real cool feature to provide to the user especially when you allow them to preview multiple zones.
         I spent time investigating some of the issues I had found yesterday
         I did get Netbook and my testing focus is Mutro / Deskpro on Netbook
         I reported issues found on Mutro (Netbook) to Jira
         Dev 2 & I had a bug triage meeting scheduled and this shall be a daily activity from today. We discussed and scrubbed the issues reported. This activity is definitely helpful.
         I also did test Welcome Bravo on Deskpro launched from nLive and shall investigate the same on Netbook before logging it in Jira.
         Some more security issues to be reported for Infra team
         Had a lunch meeting with Admin Personnel on Infrastructure related issues. I learnt that migration will happen to Windows Server 2008. Admin Personnel communicated that he would give me a test login that should help me identify risks and re-test security for Server 2008.
         Wider & Deeper coverage of Mutro on Netbook with focus towards any kind of issues that might impact the business or user experience.

-- Pradeep Soundararajan

On Mon, Sep 27, 2010 at 7:34 PM, Pradeep Soundararajan <PSoundararajan@Bravo.com> wrote:
Hi All,


         As said in Day5 report, I started with reporting the issues I had found on Friday which consumed 1/3rd of the time in day.
         The netbook had some issues and later was being fixed by Dev 2 so I still haven't started testing on it but the good thing is I didn't keep mum.
         I continued testing Deskpro from my own laptop this time.
         I continued performing security & functional tests and found a bunch of issues again which would go to Jira soon.

Strategy suggestion:

         Today's testing revealed some more security issues related to Deskpro. I could traverse to get through the Scheduled tasks set on Sriver 11 and could set tasks of my own. For instance, I could see "Orphan Port Delete" scheduled task and its frequency.
         I could get an option to look at the Windows Terminal Network which is connected to the Sriver11 although I couldn't see beyond Sriver11 on it.
         I could access Microsoft Outlook on Sriver11, set accounts on it or send and recieve emails.

         I feel, I have found enough security issues to help the teams rethink about the policies, firewall and system settings.
         Otherwise it would be same kind of information over loaded.
         I think I should focus on other quality criteria henceforth - Functionality & Usability with less focus on Security.
         That way, I can be of more value to you.
         I am open to suggestions from you though.

         Report issues found today with screenshots
         Should be getting the netbook and testing Mutro & Deskpro from that.
         Test for Usability & Functionality

On Mon, Sep 27, 2010 at 9:54 AM, Pradeep Soundararajan <PSoundararajan@Bravo.com> wrote:
Hi All,

         The report I had to send on Friday evening is what I am sending now.

         Was covering User Experience Testing of Mutro on Cloudbox
         Found issues related to User Experience & Flows that users are more likely to hit upon
         Focus then shifted to Penetration testing on Deskpro
         Some of the cool security issues were uncovered such as there are competing anti spyware programs that can be installed which gives us access to kill McAfee and take much more control over the system.
         I could uninstall Winamp, see the process running on SRIVER11, kill some of them, run a Spyware scan and so forth.
         Rest assured, these are going to Jira today.
         I shall be reporting the issues found on Friday into Jira
         I should start testing Mutro & Deskpro from Netbook
         Most testing from now on would be from the Netbook
         Focus remains on User Experience & Security

-- Pradeep Soundararajan

On Thu, Sep 23, 2010 at 9:45 PM, Pradeep Soundararajan <PSoundararajan@Bravo.com> wrote:
Hi All, 


         Today started off with a meeting with Tester where he showed me the testing he and PM had done on Mutro which gave rise to a few more ideas and scenarios for me.
         I continued to test on Cloud PC for a while tell Dev 2 came in and we decided to move to the latest version ( I was testing on RC3 )
         Post TL and Dev 2 meeting we decided that the issues I report would be considered for fixing post the Customer X/CUSTOMER Y release.
         The upgrade on cloud PC had some issues so instead of watching it get upgraded, I started picking up the issues I found yesterday, investigated and reported them.
         I was on the catch up call with CEO, TL and Dev 2.
         CEO set the goal for Black Box User Perspective Testing on Mutro
         I also continue to add on to my list of the kinds of test coverage we need to be able to achieve a wider and deeper testing.
         At about 6 PM my laptop broke (physically), I have now had a temporary fix to it which can help me continue tomorrow without any significant issues. [ pretty childish? :) ]
         I start testing Mutro from Tablet PC than Cloudbox
         I shall continue to report issues as and when I find.
         I shall log the security issues under the about to be created Security section in Jira
Help & Support:
         Joining you people mid way through your journey of Cloudbox and new customer requests, I am not the best person to be able to judge the priority of the bug, so I request the help of respective module leads to take a look at the bug reports at the end of the day and change its priority accordingly.
         I was under the impression that I was reporting the severity of the problem and not the priority because it is only you who understand the business layer who can set it.
         So, I would continue to report issues with my limited judgement and I shall rely on module leads to change the severity and priority for the issues I report.
         From my end, I shall ensure I provide as much evidence as possible to the bugs I report. 
Thanks for your time and patience that I value,

-- Pradeep Soundararajan
On Wed, Sep 22, 2010 at 7:45 PM, Pradeep Soundararajan <PSoundararajan@Bravo.com> wrote:
Hi All,

         I got access to Mutro today
         Half of my time was spent on learning & exploring Mutro & Deskpro through Mutro ( which is towards release 2.0, correct me if I am wrong )

         I reported a few issues in Jira & shall start reporting issues to respective projects henceforth.
         I found issues that breach the security policies which I shall be reporting tomorrow morning into Jira as I am on the final leg of investigation. Issues such as Bravocentre.exe and Welcome_tillSep08.exe can be copied and transported over internet and disassembled.
         TL brought to my attention of a release being made to Customer X / CUSTOMER Y next week and set a goal to find issues of Mutro from users perspective of usage that can be passed to Dev 2 ( did I spell it wrong? )
         I have attached the issues I found so far as a text file to this email ( Not to worry - these will be going to Jira ) and I am set to uncover more such issues tomorrow. 
         As a side note: I am dealing with opportunity cost of finding issues versus reporting them. One takes away time from another.
         I am making a list of things to be covered for testing of Mutro as and when I use it.
Tomorrow's plan:
         Continue testing Mutro and Deskpro for Functional, Usability & Security issues
         Report bugs in Jira
         Sit with Dev 2 once he is back and unlearn things I might have incorrectly learnt about Mutro

-- Pradeep Soundararajan

On Tue, Sep 21, 2010 at 8:03 PM, Pradeep Soundararajan <PSoundararajan@Bravo.com> wrote:
Hi All,


         Mixed day ( No power for first 2 hours and then intermittent day coupled with some good issues )
         You would discover that I am emailing from Bravo id, so I got it and Jira access too.
         I am reporting issues in Jira under QA & Testing
         Used the no electricity time to interact with Admin Personnel and understand the infrastructure and security of the Bravo cloud a little deeper
         Used the tea break to discuss with Marketing & Sales team about existing issues.
         I have planned to do a paired testing approach with Marketing and Admin Personnel individually to be able to identify more issues.
         Post TL's meeting with CEO, I learnt the focus doesn't need to be on Live as of now and is on Cloudbox & Deskpro or 2.0 release if I may say so.
         Post discussion with Marketing, I identified I would also need a Data card connection to be able to test from a typical wireless access modelling real user scenario.
         TL is going to help me get a Cloudbox tomorrow and that would also be a part of my testing from tomorrow.
         Read the Bravo Blog

         I have reported about 10 issues in Jira as of today coupled with testing performed.
         Some issues from yesterday's investigation and some with todays. I am covering security and purpose hand in hand while learning more about the product.

         Waiting to get my hands on the Cloudbox & continue to increase the test coverage
Thanks for your time so far,

-- Pradeep Soundararajan