Reflection .NET API
Walkthrough / Retrieve and Enter Data / Screen Scrape Data
In This Topic
    Screen Scrape Data
    In This Topic

    This sample navigates to a screen that has a data table, gets the data line by line, and writes it to a comma separated value list (a .csv file).

    You can follow along with the walkthrough and add the code for each step or you can just add the required references and copy in all of the code for the Program.cs file at the end of this walkthrough.

    To screen scrape data from a host

    1. In Visual Studio, create a new Console Application project and add references for the following Reflection assemblies. (Depending on your version of Visual Studio, these can be found either on the .NET tab or under Assemblies | Extensions.)
      Attachmate.Reflection
      Attachmate.Reflection.Framework
      Attachmate.Reflection.Emulation.IbmHosts
      Attachmate.Reflection.Emulation.OpenSystems

    2. Replace all the code in the Program.cs file with the following code for the terminal you are using.

      This code starts Reflection and creates a new demo session. Then it creates an event handler for the NewScreenReady event that will be used to navigate to the appropriate screen and get the data.

      Create a new session with an event handler for the NewScreenReady event
      Copy Code
      using System;
      using System.Collections.Generic;
      using System.Text;
      using System.IO;
      using Attachmate.Reflection.Framework;
      using Attachmate.Reflection.Emulation.IbmHosts;
      using Attachmate.Reflection.UserInterface;
      namespace CreateSessionFromExistingSessionFile
      {
          class Program
          {
              
              static void Main(string[] args)
              {
                  //Start a visible instance of Reflection or get the instance running at the given channel name
                  Application app = MyReflection.CreateApplication("myWorkspace", true);
      
                  //Create a new terminal
                  IIbmTerminal terminal = (IIbmTerminal)app.CreateControl(new Guid("{09E5A1B4-0BA6-4546-A27D-FE4762B7ACE1}"));
      
                  //Handle the NewScreenReady event to navigate to the screen with the data and then get the data
                  IIbmScreen screen = terminal.Screen;
                  screen.NewScreenReady += new EventHandler(screen_NewScreenReady);
      
                  //Specify the host name or IP address and connect.
                  terminal.HostAddress = "demo:ibm3270.sim";
                  terminal.Port = 623;
                  terminal.Connect();
      
                  //Make the session visible
                  IFrame frame = (IFrame)app.GetObject("Frame");
                  frame.CreateView(terminal);
                  Console.ReadKey();
              }
          }
      }
      

      This code gets a handle to the terminal of a session running in Reflection and then navigates to the screen with the data. It calls the getData() method to screen scrape the data and save it to a comma separated list file.

      Get a handle to a terminal of a running session, navigate to a screen, and call the getTheData method
      Copy Code
      //ScreenScrapeOS gets the Terminal control of a running session.
      //Before you run this sample, make sure the session used in GetControlsByFilePath()
      //is running in a Reflection workspace.
      using System;
      using System.Collections.Generic;
      using System.Text;
      using System.IO;
      using Attachmate.Reflection.Framework;
      using Attachmate.Reflection.Emulation.OpenSystems;
      namespace ScreenScrapeOS
      {
          class Program
          {
              static void Main(string[] args)
              {
                  //Get a handle to the Reflection workspace
                  Application app = MyReflection.ActiveApplication;
        
              
                  //Make sure Reflection is running
                  if (app != null)
                  {
                      //By default, the UserProfile variable is the user's myDocuments folder
                      string sessionPath = Environment.GetEnvironmentVariable("UserProfile") + @"\Documents\Micro Focus\Reflection\demoSession.rdox";
                    
                      //Get a handle to the control identified by sessionPath
                      object[] terminals = app.GetControlsByFilePath(sessionPath);
      
                      //Make sure the session is running
                      if (terminals != null && terminals.Length > 0)
                      {
                          //Get the first terminal
                          ITerminal terminal = (ITerminal)terminals[0];
      
                          //Get a handle to the screen
                          IScreen screen = terminal.Screen;
      
                          //Go to the screen with the data and wait for it to settle
                          screen.SendKeys("demodata");
                          screen.SendControlKey(ControlKeyCode.Return);
                          screen.WaitForHostSettle(6000, 3000);
      
                          //Get the screen data and save it to a .csv file
                          getTheData(screen);
      
                          Console.ReadKey();
                      }
                      else
                          Console.WriteLine("No such control exists. Check to make sure that the session from the file is running.");
                  }
                  else
                      Console.WriteLine("Failed to get Application object.");
              }
          }
      }
      

       

    3. Add the following code to the Program class in the Program.cs file

      This code navigates to the screen with the data, gets the data and formats it  row by row, using the FormatData helper method. When an empty row is retrieved, the data is saved to a file and written to the Console.

      Handle the NewScreenReady Event to navigate to a screen and  get data
      Copy Code
       //Navigate to the screen with the data and get the data
              static void screen_NewScreenReady(object sender, EventArgs e)
              {
                  string screenID1, screenID2, screenID3;
                  IIbmScreen screen = (IIbmScreen)sender;
      
                  screenID1 = screen.GetText(1, 2, 3);
                  screenID2 = screen.GetText(1, 1, 5);
                  screenID3 = screen.GetText(1, 25, 13);
                  if (screenID1 == "ATM")
                  {
                      //Enter the username and password
                      //Use macro.password to put password in
                      screen.PutText("username", 20, 16);
                      string pWord = screen.Parent.Macro.PasswordBox("Enter anything for a password", "Password");
                      screen.PutText(pWord, 21, 16);
                      screen.SendControlKey(ControlKeyCode.Transmit);
                  }
                  else if (screenID2 == "LOGON")
                  {
                     
                      //Go to the screen with the data
                      screen.SendKeys("kayak");
                      screen.SendControlKey(ControlKeyCode.Transmit);
                  }
                  else if (screenID3 == "INTERNATIONAL")
                  {
                      //get the data line by line and format it as a comma separated list
                      string data = "";
                      string dataLine = "";
                      string path = @"c:\temp\data.csv";
      
                      //Find the starting point of the data
                      ScreenPoint firstDataPoint = screen.SearchText("Q1", 1, 1, FindOption.Forward);
                      int dataRow = firstDataPoint.Row;
      
                      //Adjust column for relative position of Q1 text
                      int dataCol = firstDataPoint.Column - 1;
      
                      //Get the text and format it row by row until an empty line is retrieved
                      do
                      {
                          dataLine = screen.GetText(dataRow, dataCol, 50).Trim();
                          data += FormatData(dataLine) + Environment.NewLine;
                          dataRow++;
                      } while (String.IsNullOrWhiteSpace(dataLine) == false);
      
                      //Write the data to the console and a .csv file
                      Console.WriteLine(data);
                      File.WriteAllText(path, data);
                  }
              }
      

      This code gets the data and formats it  row by row, using the FormatData helper method. When an empty row is retrieved, the data is saved to a file and written to the Console.

      Get the data from the host screen
      Copy Code
              static void getTheData(IScreen screen)
              {
                  string data = "";
                  string dataLine = "";
                  string path = @"c:\temp\dataOS.csv";//File to write data to
      
                  //Get the screen coordinates of the first row of data
                  ScreenPoint startingRow = screen.SearchText("Jan", 1, 1, FindOptions.Forward);
                  int dataRow = startingRow.Row;
                  int dataCol = startingRow.Column;
      
                  //Get the data line by line and format it as a comma separated list
                  do
                  {
                      //Get a line of data
                      dataLine = screen.GetText(dataRow, dataCol, 50).Trim();
                     
                      //Add the line to the data string
                      data += FormatData(dataLine) + Environment.NewLine;
                      dataRow++;
      
                      //Continue until there is no data
                  } while (String.IsNullOrWhiteSpace(dataLine) == false);
                 
                 
                  //Display the data on the console
                  Console.WriteLine(data);
      
                  //Write the data to the comma separated list file
                  File.WriteAllText(path, data);
              }
      
    4. Add the FormatData method to the Program class in the Program.cs file. This method cleans up each row of data and formats it for a comma separated list.
      Format the data for a csv file
      Copy Code
              public static string FormatData(string sToRemove)
              {
                  //remove the extra spaces between numbers
                  sToRemove = sToRemove.Replace(" ", "[]");
                  sToRemove = sToRemove.Replace("][", "");
                  sToRemove = sToRemove.Replace("[]", " ");
      
                  //remove commas in numbers and then replace spaces that act
                  //as delimiters with commas to create a comma separated list
                  sToRemove = sToRemove.Replace(",", "");
                  sToRemove = sToRemove.Replace(" ", ",");
                  return sToRemove;
              }
      
      Format the data for a .csv file
      Copy Code
              static string FormatData(string dataLine)
              {
                  //Replace delimiters with commas and remove the delimiter at the end of the line
                  dataLine = dataLine.Replace("|", ",");
                  int lastComma = dataLine.LastIndexOf(",") - 1;
                  if (lastComma > 0)
                  {
                      dataLine = dataLine.Substring(0, lastComma);
                  }
                  return dataLine;
              }
      

    To test this project

    1. If you are using an Open Systems terminal, create a new VT session using a host name of "demo:UNIX" and save it in the default location (...Documents\Micro Focus\Reflection) as demoSession.rdox. Then open the session in Reflection and log in using any credentials. 
    2. Press F5 to run the project and verify that the data from the session is written to the Console.
    3. Open the c:\temp\dataOS.csv file and verify that the data from the screen is in the file.

     

    The full sample for the Program.cs file

    This is the full sample for this walkthrough.

    Full Code sample for Screen Scrape Data walkthrough
    Copy Code
    using System;
    using System.Collections.Generic;
    using System.Text;
    using System.IO;
    using Attachmate.Reflection.Framework;
    using Attachmate.Reflection.Emulation.IbmHosts;
    using Attachmate.Reflection.UserInterface;
    namespace CreateSessionFromExistingSessionFile
    {
        class Program
        {
           
            //FormatData cleans up the data and formats it as a comma separated list
            public static string FormatData(string sToRemove)
            {
                //remove the extra spaces between numbers
                sToRemove = sToRemove.Replace(" ", "[]");
                sToRemove = sToRemove.Replace("][", "");
                sToRemove = sToRemove.Replace("[]", " ");
    
                //remove commas in numbers and then replace spaces that act
                //as delimiters with commas to create a comma separated list
                sToRemove = sToRemove.Replace(",", "");
                sToRemove = sToRemove.Replace(" ", ",");
                return sToRemove;
            }
    
            //Navigate to the screen with the data and get the data
            static void screen_NewScreenReady(object sender, EventArgs e)
            {
                string screenID1, screenID2, screenID3;
                IIbmScreen screen = (IIbmScreen)sender;
                //
                screenID1 = screen.GetText(1, 2, 3);
                screenID2 = screen.GetText(1, 1, 5);
                screenID3 = screen.GetText(1, 25, 13);
                if (screenID1 == "ATM")
                {
                    //Enter the username and password
                    //Use macro.password to put password in
                    screen.PutText("username", 20, 16);
                    string pWord = screen.Parent.Macro.PasswordBox("Enter anything for a password", "Password");
                    screen.PutText(pWord, 21, 16);
                    screen.SendControlKey(ControlKeyCode.Transmit);
                }
                else if (screenID2 == "LOGON")
                {
                    //Go to the screen with the data
                    screen.SendKeys("kayak");
                    screen.SendControlKey(ControlKeyCode.Transmit);
                }
                else if (screenID3 == "INTERNATIONAL")
                {
                    //get the data line by line and format it as a comma separated list
                    string data = "";
                    string dataLine = "";
                    string path = @"c:\temp\data.csv";
    
                    //Find the starting point of the data
                    ScreenPoint firstDataPoint = screen.SearchText("Q1", 1, 1, FindOption.Forward);
                    int dataRow = firstDataPoint.Row;
    
                    //Adjust column for relative position of Q1 text
                    int dataCol = firstDataPoint.Column - 1;
    
                    //Get the text and format it row by row until an empty line is retrieved
                    do
                    {
                        dataLine = screen.GetText(dataRow, dataCol, 50).Trim();
                        data += FormatData(dataLine) + Environment.NewLine;
                        dataRow++;
                    } while (String.IsNullOrWhiteSpace(dataLine) == false);
    
                    //Write the data to the console and a .csv file
                    Console.WriteLine(data);
                    File.WriteAllText(path, data);
                }
            }
            static void Main(string[] args)
            {
                //Start a visible instance of Reflection or get the instance running at the given channel name
                Application app = MyReflection.CreateApplication("myWorkspace", true);
    
                //Create a new terminal
                IIbmTerminal terminal = (IIbmTerminal)app.CreateControl(new Guid("{09E5A1B4-0BA6-4546-A27D-FE4762B7ACE1}"));
    
                //Handle the NewScreenReady event to navigate to the screen with the data and then get the data
                IIbmScreen screen = terminal.Screen;
                screen.NewScreenReady += new EventHandler(screen_NewScreenReady);
    
                //Specify the host name or IP address and connect.
                terminal.HostAddress = "demo:ibm3270.sim";
                terminal.Port = 623;
                terminal.Connect();
    
                //Make the session visible
                IFrame frame = (IFrame)app.GetObject("Frame");
                frame.CreateView(terminal);
                Console.ReadKey();
            }
        }
    }
    
    Full Code sample for Screen Scrape Data walkthrough
    Copy Code
    //ScreenScrapeOS gets the Terminal control of a running session.
    //Before you run this sample, make sure the session used in GetControlsByFilePath()
    //is running in a Reflection workspace.
    using System;
    using System.Collections.Generic;
    using System.Text;
    using System.IO;
    using Attachmate.Reflection.Framework;
    using Attachmate.Reflection.Emulation.OpenSystems;
    namespace ScreenScrapeOS
    {
        class Program
        {
            static string FormatData(string dataLine)
            {
                //Replace delimiters with commas and remove the delimiter at the end of the line
                dataLine = dataLine.Replace("|", ",");
                int lastComma = dataLine.LastIndexOf(",") - 1;
                if (lastComma > 0)
                {
                    dataLine = dataLine.Substring(0, lastComma);
                }
                return dataLine;
            }
           
            static void getTheData(IScreen screen)
            {
                string data = "";
                string dataLine = "";
                string path = @"c:\temp\dataOS.csv";//File to write data to
    
                //Get the screen coordinates of the first row of data
                ScreenPoint startingRow = screen.SearchText("Jan", 1, 1, FindOptions.Forward);
                int dataRow = startingRow.Row;
                int dataCol = startingRow.Column;
    
                //Get the data line by line and format it as a comma separated list
                do
                {
                    //Get a line of data
                    dataLine = screen.GetText(dataRow, dataCol, 50).Trim();
                   
                    //Add the line to the data string
                    data += FormatData(dataLine) + Environment.NewLine;
                    dataRow++;
    
                    //Continue until there is no data
                } while (String.IsNullOrWhiteSpace(dataLine) == false);
               
               
                //Display the data on the console
                Console.WriteLine(data);
    
                //Write the data to the comma separated list file
                File.WriteAllText(path, data);
            }
            static void Main(string[] args)
            {
                //Get a handle to the Reflection workspace
                Application app = MyReflection.ActiveApplication;
      
            
                //Make sure Reflection is running
                if (app != null)
                {
                    //By default, the UserProfile variable is the user's myDocuments folder
                    string sessionPath = Environment.GetEnvironmentVariable("UserProfile") + @"\Documents\Micro Focus\Reflection\demoSession.rdox";
                  
                    //Get a handle to the control identified by sessionPath
                    object[] terminals = app.GetControlsByFilePath(sessionPath);
    
                    //Make sure the session is running
                    if (terminals != null && terminals.Length > 0)
                    {
                        //Get the first terminal
                        ITerminal terminal = (ITerminal)terminals[0];
    
                        //Get a handle to the screen
                        IScreen screen = terminal.Screen;
    
                        //Go to the screen with the data and wait for it to settle
                        screen.SendKeys("demodata");
                        screen.SendControlKey(ControlKeyCode.Return);
                        screen.WaitForHostSettle(6000, 3000);
    
                        //Get the screen data and save it to a .csv file
                        getTheData(screen);
    
                        Console.ReadKey();
                    }
                    else
                        Console.WriteLine("No such control exists. Check to make sure that the session from the file is running.");
                }
                else
                    Console.WriteLine("Failed to get Application object.");
            }
        }
    }