Top 10 Tips For Testing iPhone Applications

1 Accurately report available memory. 

Many of the non-reproducible bugs you run into when testing iPhone apps are related to memory problems. 

It's critical that you know and report available free memory before launching an application. 

In all likelihood, the reproducibility of a crashing iPhone app bug is related to low memory conditions. 

Consequently, a crashing defect may disappear when there's plenty of free memory.

In a previous article (Using Memory Sweep for iPhone App Testing) we described a tool that can be used to determine free memory.

2 Provide 'crash reporter' logs with your defect reports.

Each time an iPhone application crashes, a .crash file is created on your iPhone. 

You can retrieve this file when you synch your iPhone with iTunes. 

Here's a link that describes where those files are stored: iPhone Crash Logs

            3 Spy on the app from the console. 

iPhone apps will report application and system level warnings to the console.

You can view these warnings in real-time using Apple's iPhone Configuration Tool. 

By knowing what's being reported when interacting with an app can help you refine the steps you need to reproduce tricky (and memory related) problems.

        4 Test under low memory conditions. 

This relates to #1 above.  You'll be able to tease additional crashing bugs if you force free memory to a very low level, e.g. < 2MB, before proceeding with your tests. 

One way to do this is to open several Safari windows before you start your testing.

            5 Screenshots, screenshots, screenshots.

Nothing makes a UI bug stand out for a developer than when you send screenshots. 

And with the iPhone's built-in screen capture, there's no excuse not to do this.

6  Provide useful defect characterization information.

Developers always like to have help in their debugging process, and useful defect characterization helps them narrow down the root cause of a bug. 

If a crash happens under low memory conditions (see #1 and #4 above), then try it under conditions where there's lots of memory available, e.g. >40MB.  If a problem occurs under iPhone OS 2.2.x, then try it under 3.x.

            7 Create connectivity problems.

If you're testing an iPhone app that depends on internet connectivity, then test for degraded or unavailable connectivity.

It's easy to make connectivity unavailable by simply turning on Airplane Mode.

To degrade connectivity, especially on Edge or 3G, employ some sort of metallic "shield" on top of your iPhone.

            8 Boundary test data input.

Wherever an iPhone app asks for text input, you have an opportunity to find a bug. 

My favorite technique for this is to copy a huge amount of text, then paste it into each text field. 

You'd be surprised at how this trips up some apps.

Additionally, we’ve been finding that application errors are generating when entering the following characters into text fields: !@#$%^&*()_.

(Note: Holding down letters (A, E, I, O, etc) or symbols ($, !, &, etc) on the onscreen keyboard generates a keyboard popup that includes localized and 2-byte characters. These should also be entered into text fields.)

            9 Gather up UDIDs (unique device identifiers) early. 

This is a simple logistics task but seems to be one that becomes critical as the first build approaches.

And it's a hassle for the dev team to add new UDIDs and create new provisioning files as each new person wants to install an application during development. 

Get the UDIDs of all know devices that will be used during testing and set a cut-off date for the addition of any new devices. 

Check out Find UDID with a single click. You can also connect all your iPhones and iPods touches to your computer and use the iPhone Configuration Tool to collect UDIDs.

10  Employ background applications.

But the iPhone can only have one application running at a time, right? That's true for those of us that develop and test applications, but not for Apple. 

Applications that continue running in the background on the iPhone are Safari, iPod and Mail. 

And what about reminders and push notifications?  These "interrupters" can affect the behavior of an application under test.

Also, since iPhones and iPod touches are devices that users buy primarily to use as a phone or a music player, it’s important to test that an app can gracefully handle situations where the user receives a call or plays music from their music library (iTunes) while the app is running. We’ve seen issues here where apps aren’t smoothly multi-functioning in these areas.

Standard Checklist for Mobile Applications

Sr.No	Description/Steps	Expected Output
1	Installation “.exe”
1.1	Install the .exe file.	1.      Application name along with its version and a small logo as per suitable should display on the installation window bar.
		2.      At the time of installation progress bar    always display there.
		3.      In application installation check box show for agree the terms & condition and there back & next options should display.
		4.      Application logo should be appeared on the  application menu screen with name.
2	General Text Box
2.1	Check display on form.	1. Form Title, Name, Labels, Number and mandatory marks should be as per requirement.
		2. All mandatory marks should present.
3	General Text Box
3.1	Check the default value.	It is a free text box, it should be blank.
3.2	Try to enter leading spaces in the text box.	Input should not be allowed by the system.
3.3	Enter ‘abcde      ‘in the text box. Click Submit.	While saving the Input in the database system should trim the trailing spaces.
3.4	Enter invalid data for each field.	There should be display validation message for incorrect data.
3.5	Enter more data than required.	There should be maximum length defined.

Tracking iPhone Memory Leaks

http://www.streamingcolour.com/blog/tutorials/tracking-iphone-memor...

What is a memory leak and why should I care?

A memory leak is when your program loses track of a piece of memory that was allocated. The consequence is that the “leaked” memory will never be freed by the program. This usually happens when a piece of code does a “new” or a “malloc” (or an “alloc” in Objective-C) but never does a corresponding “delete”, “free” or “release”, respectively.

When you new, malloc, or alloc, what the Operating System is doing is giving your program a chunk of memory on the heap. The OS is saying, “here, have this block of memory at this memory address.” It expects you to hold a reference to that memory address (usually in the form of a pointer) and it’s relying on you to tell the OS when you’re done with it (by calling free, delete, or release).

Memory leaks happen when you throw away your pointer to that memory. If your program no longer knows where on the heap your memory is allocated, how can you ever free it?

So why should you care? In the most minor case, you’re wasting memory that will be freed when the user quits your app. In the worst case, you could have a memory leak that happens every frame. That’s a good way to end up crashing your program, especially if a user lets it run for a long time. (Note: in iOS you can’t leak system-wide memory. In the worst-case you will cause your app to run out of memory and crash, but you can’t bring the whole system down).

For more general information on memory leaks, have a look at wikipedia for a start:

http://en.wikipedia.org/wiki/Memory_leak

How do I know I’ve got a memory leak?

Some memory leaks are easy to see by looking at your code. Some are much more difficult. This is where Instruments comes in. Instruments has a “Leaks” tool that will tell you exactly where you’re leaking memory so that you can get in there and fix it!

An Example App

I’ve created an example application that leaks memory in two places. The code is as follows (note: I’ve wrapped some strings so that they show up in the space available on the site, this code won’t compile unless you fix that if you copy it into XCode):

And there’s a LeakedObject class, but the code isn’t relevant to the example. I’m going to go ahead a build my InstrumentsTest iPhone app in Debug and get it running on my iPhone. Once I’ve done that, I’ll boot up Instruments (typing “Instruments” into Spotlight should find it).

Instruments

When you launch Instruments, you should be given the chance to select from a variety of different Instrument tools to use. On the left-hand side, choose iPhone. On the right-hand menu, double-click the “Leaks” tool:

Once you’ve done that, you should see a window that looks like this:

Make sure your iPhone is still connected to your computer. In the top-left corner of the window you’ll see a drop-down menu that says “Launch Executable”. Click on that and make sure that your iPhone (not your computer) is selected as the active device. Then scroll down to “Launch Executable” and you should see a list of all the apps that are installed on your iPhone. Find the app that you want to run “Leaks” on (in this case, InstrumentsTest) and click it.

You’re now ready to go. Click on the red “Record” button:

and it will launch your application for you and start recording every memory allocation you do. It will also automatically check for leaks every 10 seconds. You can change how often the automatic leak check runs, or you can set it to run only when you tell it too (when it checks for leaks, the entire app will freeze for about 3-5 seconds, so it can be annoying if you’re trying to play test and check for leaks at the same time). What I usually do is set it to manual control, and then hit the “Check for leaks” button whenever I need to (e.g. after loading a new game mode, after quitting the game back to the main menu, etc). However, for this example, I’m going to leave it on auto:

After the app has run for a few seconds, the auto leak check will run and lo and behold, it has found two memory leaks! Fantastic! What do we do now?

Extended Detail View

Instruments is very sneaky: it doesn’t make it obvious what to do next. What you need to notice is that row of buttons along the bottom of the window. See that little one made up of two rectangles? Hover your mouse over it and it will say “Extended Detail View”. (Note: You can also open this via View -> Extended Detail)

Update (2010-09-21) – In newer versions of Xcode, the buttons along the bottom have been removed. Click the “Right Pane” icon button at the top of the window to get the Extended Detail View. It is also still available under the View menu.

Click that button and a window opens up on the right-hand side of the screen that provides all kinds of handy details about your leaks!

Click on one of the memory leaks. The Extended Detail View will show you a complete stack trace to the point where your leaked memory was allocated. In our example above, clicking on the first leak reveals that a leak occurred inside [NSString initWithUTF8String]. If you look one step higher in the stack trace, you’ll see the last call inside my app was to [InstrumentsTestViewController viewDidLoad].

Here’s the really cool part, double-click on that line in the Extend Detail View and it opens an XCode window right to the culprit!

Update (2010-09-21) – Newer versions of Xcode will display the method name where the leak was allocated right in the main leaks view. You can double-click the leak (without having to open the Extended Details) to see the line of code that leaked. Of course, the Extended Details View is still there in case you want the full stack trace.

In this case we see that it was the first NSString allocation that leaked. Here’s where you need to do a bit of detective work. This is an extremely simple case, but it can get more tricky to find out why something’s leaky. Let’s take a closer look at our example.

In viewDidLoad we allocate some memory for the string, like this:

  

And in dealloc, we release it like this:

    [mMyLeakyString release];

So your immediate reaction might be that there shouldn’t be a leak. However, let’s search the code for all references to mMyLeakyString. That turns up this line inside doSomethingNow:

Notice that we’ve allocated a new string and assigned the pointer to mMyLeakyString. The problem is that we never released mMyLeakyString before it started pointing to something else. So the original string is now floating around on the heap and we have no way of freeing that memory. What the release call inside dealloc is actually doing is freeing the 2nd string that we allocated in doSomethingNow, because that’s where the pointer is pointing.

So, to fix this, we might change doSomethingNow to something like this:

string. When you build and run your app in Instruments again, you’ll see there’s one fewer memory leak. Of course, there are probably some better ways to handle NSStrings in your project, but if you had to do it this way, this would fix it.

Let’s take a look at that second leak. Clicking on it again reveals the callstack of what led to the leak. Finding the last call inside our app shows that the leak came from inside the LeakyClass::LeakyClass() constructor:

Double-click that in the stack and it opens up the culprit in XCode again:

Here we see the constructor does a new of a LeakedObject. But what’s this? The destructor never deletes the pointer? Well that’s no good! For every new, there needs to be a corresponding delete! So let’s change the destructor to this:

// Destructor LeakyClass::~LeakyClass() {     if (mLeakedObject != NULL)     {         delete mLeakedObject;         mLeakedObject = NULL;     } }

Build and run your app through Instruments again and you should be memory leak free!

Mobile Web Testing

Let’s have first web testing checklist.
1) Functionality Testing
2) Usability testing
3) Interface testing
4) Compatibility testing
5) Performance testing
6) Security testing

1) Functionality Testing:

Test for – all the links in web pages, database connection, forms used in the web pages for submitting or getting information from user, Cookie testing.

Check all the links:

• Test the outgoing links from all the pages from specific domain under test.
• Test all internal links.
• Test links jumping on the same pages.
• Test links used to send the email to admin or other users from web pages.
• Test to check if there are any orphan pages.
• Lastly in link checking, check for broken links in all above-mentioned links.

Test forms in all pages:
Forms are the integral part of any web site. Forms are used to get information from users and to keep interaction with them. So what should be checked on these forms?

• First check all the validations on each field.
• Check for the default values of fields.
• Wrong inputs to the fields in the forms.
• Options to create forms if any, form delete, view or modify the forms.

Let’s take example of the search engine project currently I am working on, In this project we have advertiser and affiliate signup steps. Each sign up step is different but dependent on other steps. So sign up flow should get executed correctly. There are different field validations like email Ids, User financial info validations. All these validations should get checked in manual or automated web testing.

Cookies testing:

Cookies are small files stored on user machine. These are basically used to maintain the session mainly login sessions. Test the application by enabling or disabling the cookies in your browser options. Test if the cookies are encrypted before writing to user machine. If you are testing the session cookies (i.e. cookies expire after the sessions ends) check for login sessions and user stats after session end. Check effect on application security by deleting the cookies. (I will soon write separate article on cookie testing)

Validate your HTML/CSS:

If you are optimizing your site for Search engines then HTML/CSS validation is very important. Mainly validate the site for HTML syntax errors. Check if site is crawlable to different search engines.

Database testing:

Data consistency is very important in web application. Check for data integrity and errors while you edit, delete, modify the forms or do any DB related functionality.
Check if all the database queries are executing correctly, data is retrieved correctly and also updated correctly. More on database testing could be load on DB, we will address this in web load or performance testing below.

2) Usability Testing:

Test for navigation:

Navigation means how the user surfs the web pages, different controls like buttons, boxes or how user using the links on the pages to surf different pages.
Usability testing includes:
Web site should be easy to use. Instructions should be provided clearly. Check if the provided instructions are correct means whether they satisfy purpose.
Main menu should be provided on each page. It should be consistent.

Content checking: 

Content should be logical and easy to understand. Check for spelling errors. Use of dark colors annoys users and should not be used in site theme. You can follow some standards that are used for web page and content building. These are common accepted standards like as I mentioned above about annoying colors, fonts, frames etc.
Content should be meaningful. All the anchor text links should be working properly. Images should be placed properly with proper sizes.
These are some basic standards that should be followed in web development. Your task is to validate all for UI testing

Other user information for user help:

Like search option, sitemap, help files etc. Sitemap should be present with all the links in web sites with proper tree view of navigation. Check for all links on the sitemap.
“Search in the site” option will help users to find content pages they are looking for easily and quickly. These are all optional items and if present should be validated.

3) Interface Testing:

The main interfaces are:
Web server and application server interface
Application server and Database server interface.

Check if all the interactions between these servers are executed properly. Errors are handled properly. If database or web server returns any error message for any query by application server then application server should catch and display these error messages appropriately to users. Check what happens if user interrupts any transaction in-between? Check what happens if connection to web server is reset in between?

4) Compatibility Testing:

Compatibility of your web site is very important testing aspect. See which compatibility test to be executed:

• Browser compatibility
• Operating system compatibility
• Mobile browsing
• Printing options

Browser compatibility:

In my web-testing career I have experienced this as most influencing part on web site testing.
Some applications are very dependent on browsers. Different browsers have different configurations and settings that your web page should be compatible with. Your web site coding should be cross browser platform compatible. If you are using java scripts or AJAX calls for UI functionality, performing security checks or validations then give more stress on browser compatibility testing of your web application.
Test web application on different browsers like Internet explorer, Firefox, Netscape navigator, AOL, Safari, Opera browsers with different versions.

OS compatibility:

Some functionality in your web application is may not be compatible with all operating systems. All new technologies used in web development like graphics designs, interface calls like different API’s may not be available in all Operating Systems.
Test your web application on different operating systems like Windows, Unix, MAC, Linux, Solaris with different OS flavors.

Mobile browsing:

This is new technology age. So in future Mobile browsing will rock. Test your web pages on mobile browsers. Compatibility issues may be there on mobile.

Printing options:

If you are giving page-printing options then make sure fonts, page alignment, page graphics getting printed properly. Pages should be fit to paper size or as per the size mentioned in printing option.

5) Performance testing:

Web application should sustain to heavy load. Web performance testing should include:
Web Load Testing
Web Stress Testing

Test application performance on different internet connection speed.

In web load testing test if many users are accessing or requesting the same page. Can system sustain in peak load times? Site should handle many simultaneous user requests, large input data from users, Simultaneous connection to DB, heavy load on specific pages etc.

Stress testing: Generally stress means stretching the system beyond its specification limits. Web stress testing is performed to break the site by giving stress and checked how system reacts to stress and how system recovers from crashes.
Stress is generally given on input fields, login and sign up areas.

In web performance testing web site functionality on different operating systems, different hardware platforms is checked for software, hardware memory leakage errors,

6) Security Testing:

Following are some test cases for web security testing:

• Test by pasting internal url directly into browser address bar without login. Internal pages should not open.
• If you are logged in using username and password and browsing internal pages then try changing url options directly. I.e. If you are checking some publisher site statistics with publisher site ID= 123. Try directly changing the url site ID parameter to different site ID which is not related to logged in user. Access should denied for this user to view others stats.
• Try some invalid inputs in input fields like login username, password, input text boxes. Check the system reaction on all invalid inputs.
• Web directories or files should not be accessible directly unless given download option.
• Test the CAPTCHA for automates scripts logins.
• Test if SSL is used for security measures. If used proper message should get displayed when user switch from non-secure http:// pages to secure https:// pages and vice versa.
• All transactions, error messages, security breach attempts should get logged in log files somewhere on web server.

 thanks,

kalyan.